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Abstract 

The  Air  Force  and  Department  of  Defense  (DoD)  tend  to  implement  user  based  IT 
systems  without  quantifying  whether  those  systems  would  be  properly  utilized  by  the 
target  populous.  Focus  is  generally  emphasized  on  mission  enhancement  rather  than 
looking  at  how  or  if  it  will  be  utilized  by  organizations.  There  is  no  reason  to  implement 
cloud  computing  with  the  same  disregard  for  acceptance  and  success.  The  day  of  large 
amounts  of  data  is  here  and  needs  to  converge  with  what  this  thesis  investigates,  the 
factors  that  positively  influence  organization  acceptance  and  success  of  cloud  computing 
specifically  in  the  DoD  so  that  is  can  properly  maintain,  utilize  and  store  that  data.  The 
research  focused  on  that  utilization  and  will  better  prepare  the  system  engineers  to  ensure 
the  minimum  amount  of  time  for  "total"  implementation  and  utilization.  An  in-depth 
analysis  was  conducted  to  clarify  the  effects  of  cloud  on  organizational  success  in  the 
DoD.  The  model  developed  from  this  research  quantified  acceptance  and  success  in 
regard  to  the  implementation  of  cloud  computing.  The  model  is  based  on  success  due  to 
"business  model"  factors  discovered  during  a  Delphi  study  of  industry  and  DoD  experts 
(Okoli,  2004).  One  of  the  chief  concerns  is  that  if  this  technology  is  fielded  without 
addressing  whether  and  how  the  organizations  will  utilize  it,  then  it  will  flounder  without 
being  used  as  expected,  or  worse  yet,  could  become  a  failed  technology  with  respect  to 
the  direction  the  DoD  intended.  Therefore,  a  focus  on  the  factors  affecting  acceptance 
and  success  could  ultimately  inhibit  or  influence  that  direction  and  help  us  to  make  better 
use  of  this  new  technology. 
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CLOUD  COMPUTING  IMPLEMENTATION  ORGANIZATIONAL  SUCCESS  IN 


THE  DEPARTMENT  OF  DEFENSE 


I.  Introduction 


"The  shift  to  "light  technologies, "  that  is,  cloud  services,  which  can  be  deployed 
rapidly,  and  shared  solutions  will  result  in  substantial  cost  savings,  cdlowing  agencies  to 
optimize  spending,  and  allowing  agencies  to  reinvest  in  their  most  critical  mission  needs. 
Agencies  must  focus  on  consolidating  existing  data  centers,  reducing  the  need  for 
infrastructure  growth  by  implementing  a  "Cloud  First"  policy  for  services,  and 
increasing  their  use  of  available  cloud  and  shared  sendees "  (Vivek  Kundra,  U.S.  Chief 
Information  Officer). 

Overview 

This  section  identifies  numerous  benefits  as  well  as  the  challenges  that  are  faced 
when  operationalizing  a  new  technology.  It  also  identifies  the  implementation  obstacles 
that  have  been  encountered  in  the  past  and  how  those  obstacles  can  be  overcome.  Next, 
this  section  describes  how  far  behind  industry  DoD  is,  in  regards  to  cloud  computing. 
Finally,  this  section  establishes  the  research  objectives,  research  questions,  and  the 
organization  of  this  thesis. 


Benefits 

The  many  benefits  of  cloud  computing  research  that  the  DoD  may  experience 
include  advancement  in  implementation,  efficiency  and  effectiveness.  Those  benefits, 
listed  below,  are  further  elaborated  in  Section  II  of  this  paper. 

•  Continuous  Refresh  •  Improved  Mission  Focus 

•  Rapid  Elasticity  •  Lower  Barriers  to  Entry 
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Challenges 


This  study  will  show  how  to  overcome  the  many  challenges  that  the  DoD  faces 
when  implementing  a  new  technological  solution  such  as  cloud  computing.  The  Air 
Force  and  DoD  tend  to  implement  Information  Technology  (IT)  systems  without 
quantifying  whether  those  systems  would  be  properly  utilized  by  the  target  populous.  The 
emphasis  is  on  mission  enhancement,  rather  than  how  or  if  it  will  be  utilized  by 
organizations.  There  is  no  reason  to  assume  cloud  computing  should  be  implemented 
with  the  same  disregard  for  acceptance  and  success.  The  day  of  Big  Data  is  here,  and 
should  converge  ("Big  Data  in  the  Cloud,"  2013)  with  the  factors  that  positively  influence 
organizational  acceptance  and  success  of  cloud  computing,  specifically  in  the  DoD,  so 
that  data  can  be  properly  maintained,  utilized  and  stored,  which  this  thesis  will 
investigate.  Big  Data  was  defined  by  ISACA  (Information  Systems  Audit  and  Control 
Association)  as: 

" ...data  sets  that — due  to  their  size  (volume),  the  speed  they  are  created  with 
(velocity),  and  the  type  of  information  they  contain  (variety) — are  pushing  the  existing 
infrastructure  to  its  limits"  (Mario  Bojilov,  President,  Board  of  ISACA-Brisbane). 

The  thesis  research  focused  on  utilization  and  will  better  prepare  system  engineers 
to  ensure  minimum  time  for  "total"  implementation  and  utilization.  An  in-depth  analysis 
was  conducted  to  clarify  the  effects  of  cloud  computing  on  organizational  success  in  the 
DoD.  The  models  developed  from  this  research  quantifies  acceptance  and  success  in 
regard  to  the  implementation  of  cloud  computing.  The  models  are  based  "business  model 
factors”  discovered  during  a  Delphi  study  of  industry  and  DoD  experts  (Okoli,  2004). 
One  of  the  chief  concerns  is  if  this  technology  is  fielded  without  addressing  whether  and 
how  the  organizations  will  utilize  it,  then  it  will  flounder  without  being  used  as  expected; 
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worse,  it  could  become  a  failed  technology  with  respect  to  the  direction  the  DoD 
intended.  Therefore,  a  focus  on  the  factors  affecting  acceptance  and  success  could 
ultimately  inhibit  or  influence  that  direction  and  help  the  DoD  make  better  use  of  this 
new  technology. 

Implementation  Problems 

Technology  acceptance  theories  suggest  there  are  could  be  implementation 
obstacles  when  instituting  cloud  computing,  indicating  the  importance  of  this  research.  If 
DoD  organizations  adopt  tools  identified  in  this  research  when  implementing  a  cloud 
solution,  many  of  these  hurdles  could  be  lowered  to  a  manageable  level. 

The  challenges  that  exist  in  industry  include  control,  security  and  privacy,  costs, 
vendor  standards,  transparency,  and  reliability.  Control  is  critical  to  ensuring  that  users 
have  the  ability  to  adjust  the  system  design  based  on  changing  requirements.  Security  and 
privacy,  identified  as  critical  by  DoD  experts  as  a  part  of  this  research,  are  also  important 
in  industry.  A  survey  of  chief  information  officers  and  IT  executives  by  International 
Data  Corporation  found  that  75%  of  participants  rated  security  as  their  top  priority;  this 
reflects  the  importance  of  security  and  privacy.  Security  in  all  cloud  models  has  been 
found  to  affect  accessibility,  reliability  and  overall  access  to  a  cloud  solution  (Subashini 
et  al.,  2010).  Costs,  relative  to  bandwidth,  can  negate  any  financial  advantages  to  cloud 
computing  and  should  be  closely  analyzed  to  prevent  cost  overruns  due  to  over 
estimating  of  needs.  The  amount  of  vendors  in  the  cloud  landscapes  enables  a  disarray  of 
locked-in  standards.  As  new  technologies  go,  these  standards  will  be  slowly  achieved. 
The  vastness  of  access  to  the  cloud  creates  a  transparency  issue  for  companies  when 
trying  to  metric  this  access  to  a  prospective  client.  Continuous  access  creates  a  reliability 
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issue  with  new  technology,  and  cloud  solutions  are  no  different.  These  obstacles  in 
industry  translate  directly  to  DoD  implementation  and  further  research  will  cement  that 
idea  (Leavitt,  2009). 

The  DoD  Cloud  Computing  Strategy  dated  July  5,  2012,  identifies  the  DoD 
strategy  regarding  any  type  of  implementation.  The  driving  factor  behind  how 
implementation  will  take  place  pertains  to  how  it  will  benefit  the  Joint  Information 
Environment  (JIE)  through  increased  mission  effectiveness  and  operational  efficiencies 
(DoD,  2012). 

The  Defense  Information  Systems  Agency  (DISA)  is  the  DoD  communications 
arm  that  carries  the  bulk  of  the  workload  in  implementing  new  IT  endeavors.  They  have 
not  identified  the  type  of  strategy  they  plan  to  use  and  will  follow  industry  guidance  upon 
cloud  implementation  (Cloud  Broker  RFI,  2012).  This  research  could  be  tailored  to  that 
implementation  as  needed. 

Research  Objectives 

The  following  excerpts  emphasize  the  importance  of  DISA  striving  to  achieve 
efficient  and  successful  implementation  of  a  cloud  solution.  The  fact  that  they  have  been 
identified  as  the  DoD  provider  of  that  solution  further  proves  that  philosophy. 

DoD  News  Release  identifying  DISA  as  the  cloud  service  broker: 

"DISA  has  been  named  as  the  enterprise  cloud  sendee  broker  to  help  maintain 
mission  assurance  and  information  interoperability  within  this  new  strategy"  ( DoD  New 
Release,  2012). 

DoD  Chief  Information  Officer  (CIO)  Cloud  Computing  Strategy  guidance: 

"Implement  cloud  computing  as  the  means  to  deliver  the  most  innovative, 
efficient,  and  secure  information  and  IT  sendees  in  support  of  the  Department' s  mission, 
anywhere,  anytime,  on  any  authorized  device"  (DoD  Chief  Information  Officer,  2012). 
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As  cloud  computing  is  a  new  technological  endeavor  in  the  Air  Force,  the 
benefits,  challenges,  and  implementation  problems,  make  successful  implementation  of 
cloud  computing  as  an  IT  solution  paramount  to  future  mission  achievement.  The  role  to 
execute  cloud  computing  brokerage  has  been  undertaken  by  DISA  and  based  on  DISA's 
factors  of  "success"  ("GO  Cloud  Broker,"  2012),  this  research  will  create  a  model 
generated  from  a  Delphi  study  of  DoD  and  industry  experts.  The  ultimate  goal  of  this 
research  is  a  viable  cloud  computing  implementation  model  to  use  for  successful 
execution. 

This  research  will  provide  background  into  cloud  computing,  to  include 
definition,  characteristics,  and  benefits.  More  importantly,  this  research  shows  what 
organizational  "business  associated"  factors  should  be  realized  by  DISA  before  the 
brokerage  solution  is  implemented.  By  DISA  addressing  these  specific  factors,  they  can 
adjust  the  implementation  strategy  to  ensure  they  are  mitigated.  The  research  models 
developed  could  also  be  used  DoD-wide  for  future  cloud  implementation.  At  the 
conclusion  of  this  research,  recommendations  will  be  made,  based  on  the  Delphi  study 
feedback,  to  aid  DISA  in  its  cloud  brokerage  endeavor  providing  a  viable  implementation 
model  to  follow. 

Based  on  the  DISA  COA  (Course  Of  Action)  brief  ("GO  Cloud  Broker,"  2012), 
feedback  from  DISA  via  teleconference  and  e-mail  the  following  are  the  basis  for 
successful  implementation: 

•  Actual  availability  of  the  data  and  system  compared  with  Sendee  Level 

Agreement  (SLA)  requirements 

•  Reliability  of  the  data  and  system  compared  to  expectations 

•  Maintainability  of  data  and  the  system  compared  to  expectations 

•  Sen’iceability  -  supplier  performance  compare  with  contractual  conditions 
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•  Structure  at  the  organization 

•  Training  at  the  organization 

•  Usability  at  the  organization 

•  Profitability  through  increased  return  on  mission  effectiveness 

There  have  been  numerous  research  streams  investigating  technology  acceptance 
theories.  These  theories  will  be  used  to  support  model  development  regarding  successful 
cloud  computing  implementation  efforts  by  DISA.  TAM  (Technology  Acceptance 
Model),  TAM2  (Technology  Acceptance  Model  2),  UTAUT  (Unified  Theory  of 
Acceptance  and  Use  of  Technology),  TAM3  (Technology  Acceptance  Model  3),  and  M- 
TAG  (Model  for  Technology  Acceptance  on  Groups)  are  all  appropriate  theories  that  are 
relevant  to  this  research  and  the  models  developed.  The  theoretical  importance  of  these 
theories  and  their  applicability  to  this  research  will  be  elaborated  in  Chapter  IV. 

Research  Questions 

Data  obtained  during  this  research  will  answer  the  following  questions  regarding 
the  cloud  computing  brokerage  pursuit  by  DISA.  The  answers  from  industry  experts  will 
be  analyzed  through  Measures  of  Effectiveness  (MoEs)  and  provide  the  basis  for  the 
development  of  the  models. 

RESEARCH  QUESTION  1:  What  organizational  variables  or  processes  influence 
successful  cloud  computing  implementation  in  the  DoD? 

RESEARCH  QUESTION  2:  How  could  DISA  implement  a  cloud  solution? 

RESEARCH  QUESTION  3:  Can  a  "model"  be  developed  that  will  assist  DISA's  strategy 
for  successfully  fielding  cloud  computing  in  the  DoD? 
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Thesis  Organization 


Chapter  I  introduced  the  research  overview,  defined  successful  cloud  computing 
implementation,  identified  the  research  questions  being  answered  by  the  research,  and 
introduced  the  supporting  theories.  Chapter  II  defines  cloud  computing  in  detail  and 
explains  cloud  models,  services,  benefits,  and  characteristics  that  will  be  reviewed. 
Chapter  III  explains  that  Grounded  Theory  and  the  Delphi  method  will  be  used  to 
question  DoD  and  industry  experts  regarding  cloud  computing  implementation.  Chapter 
IV  has  the  analysis  of  the  responses,  theory  support,  analysis  of  the  variables,  and 
presents  models.  Chapter  V  contains  the  discussion,  recommendations,  and  areas  for 
future  research. 
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II.  Background 


Cloud  Computing  Defined 

The  National  Institute  of  Standards  and  Technology  (NIST)  describes  cloud 
computing  as: 

“A  model  for  enabling  ubiquitous,  convenient,  on-demand  network  access  to  a  shared 
pool  of  configurable  computing  resources  ( e.g.  networks,  servers,  storage,  applications, 
and  services)  that  can  be  rapidly  provisioned  and  released  with  minimal  management 
effort  or  service  provider  interaction.  This  cloud  model  promotes  availability  and  is 
composed  of  five  essential  characteristics,  three  service  models  and  four  deployment 
models.  ”  (Mel  and  Grance,  2011) 

This  research  does  not  target  the  technical  aspects  of  the  communications  within 
cloud  computing.  It  is  assumed  that  the  reader  possesses  this  technical  knowledge,  which 
should  enhance  the  strategic  point  of  view  this  thesis  presents.  Knowing  the  background 
should  assist  in  implementing  the  presented  models  developed  from  this  research. 

Cloud  computing  offers  an  invaluable  pathway  to  access  tremendous  amounts  of  data 
by  the  "click  of  a  button.”  The  ability  for  the  cloud  to  be  scalable,  reduce  costs,  decrease 
points  of  entry,  and  increase  organizational  competencies  has  made  it  very  attractive  to 
industry.  These  same  attractive  traits  have  led  the  DoD  to  start  the  process  of  cloud 
implementation. 

Beginning  with  the  onset  of  Amazon  and  Google  cloud  models,  the  era  of  Big  Data 
has  driven  industry  and  DoD  to  adapt  their  way  of  sharing  critical  and  non-critical 
information.  The  characteristics  of  cloud  computing  have  to  be  addressed  before 
implementing  any  type  of  solution.  The  inherent  characteristics  include  "continuous 
refresh,  lower  costs,  on-demand  self  service,  broad  network  access,  resource  pooling, 
rapid  elasticity  and  measured  service”  (Barcomb,  2009).  Four  of  these  characteristics  will 
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be  discussed  further.  Continuous  refresh,  lower  costs,  and  rapid  elasticity  are  later 
included  as  benefits. 

A  cloud  solution  also  offers  multiple  capabilities  depending  on  the  desired  effect  to 
the  organization.  Services  such  as  Infrastructure  as  a  Service  (IaaS),  Platform  as  a  Service 
(PaaS)  and  Software  as  a  Service  (SaaS)  are  all  types  of  offerings  stemming  from  a  cloud 
solution.  These  characteristics,  combined  with  one,  or  all,  of  the  services  offer  leveraged 
benefits  when  properly  deployed.  Figure  1  shows  cloud  computing  architecture. 


Figure  1 .  Cloud  Computing  Architecture 

Cloud  Computing  Platforms 

There  are  three  platforms  that  are  deployed  by  vendors  that  are  available  to  the 
consumer.  Great  care  should  be  taken  when  choosing  a  service  based  on  the  needs  of  the 
customer.  Certain  services  fulfill  different  requirements  of  the  user  at  designed  levels  of 
control  and  utilization.  Moving  to  the  lower  level  services,  the  look  and  feel  of  the 
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service  emulate  the  local  hardware  and  software  utilization  (Barcomb,  2009). 

IaaS :  The  base  service  is  offered  to  users  desiring  the  most  control  over  the 
environment.  Vendors  providing  an  IaaS  service  solely  provide  hardware,  giving  the  user 
the  flexibility  to  use  whatever  software  their  needs  require.  This  service  is  provided 
virtually,  which  limits  how  the  user  is  able  to  realize  what  hardware  is  being  used.  This 
service  should  be  selected  by  users  requiring  absolute  control  over  the  cloud. 

PaaS :  This  offers  the  ability  for  the  user  to  access  pre-programmed  applications  that 
are  provided  by  the  vendor.  The  applications  are  often  custom  coded  based  on  user 
requirements  and  transparent  to  the  user.  The  coded  application  platform  is  provided, 
operated  and  maintained  by  the  service  provider.  This  service  is  desired  when  custom 
applications  are  needed  because  commercial  solutions  available  are  not  adequate. 

SaaS:  Here,  the  provider  supplies  the  software  required  by  the  customer.  The  provider 
retains  the  responsibility  to  install,  maintain,  update,  and  operate  the  software  and 
operating  system,  as  well  as  the  associated  hardware  that  operates  the  applications.  These 
applications  are  accessed  by  the  user  remotely  from  a  thin  client  via  virtualization  of  the 
platform.  This  limits  the  user  visibility  of  the  underlying  hardware.  This  service  is 
beneficial  if  the  user  is  not  concerned  with  the  hardware,  but  rather  the  specific  software 
set  needed  for  their  operations.  At  this  time,  and  per  the  DISA  operations  section,  this  is 
the  service  being  pursued  (Barcomb,  2009). 

Cloud  Computing  Deployment  Models 

To  understand  how  DISA  would  implement  the  models  developed  from  this  research, 
there  should  be  an  understanding  of  the  different  environments  that  the  platforms  could 
be  used  in.  Ideally,  there  should  be  a  hierarchical  understanding  of  these  environments, 
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leading  to  the  model  implementation  (Figure  1).  There  are  four  different  types  of 
deployment  available  to  users  that  are  implemented  by  vendors  via  a  cloud  environment; 
Public,  Private,  Hybrid,  and  Community  (Figure  2).  They  each  offer  their  own  strengths 
and  weaknesses.  Understanding  the  model  types  and  how  they  fit  together  is  pivotal  to 
comprehension  of  the  underlying  services  that  exist  in  a  cloud  environment.  All  cloud 
services,  later  defined,  exists  within  these  deployment  types.  A  public  cloud  implements 
service  in  an  environment  where  the  vendor  is  responsible  for  all  operating  expenses.  The 
actual  services  are  provided  "publicly"  to  the  user,  external  to  the  system,  and  costs  are 
passed  to  consumers.  A  private  cloud  exists  within  the  organization  that  requests  the 
services  where  they  can  customize  as  they  see  fit.  A  hybrid  cloud  is  a  mixture  of  public 
and  private  clouds  where  organizations  only  make  part  of  their  data  available  via  a  public 
cloud,  through  the  vendor  chosen  to  offer  their  services,  and  more  critical  data  is  only 
offered  via  the  private  cloud.  A  community  cloud  exists  within  like  organizations  that 
have  established  similarities  in  their  requested  services  or  analogous  objectives 
(Barcomb,  2009). 
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Figure  2.  Cloud  Computing  Models 


Cloud  Computing  Characteristics 

Public  and  private  clouds  offer  many  advantages  to  the  IT  manager  that  decides  to 
incorporate  cloud  services  into  their  system  architecture.  Four  identified  characteristics 
are  clarified  to  explain  the  benefits  (Barcomb,  2009): 

1. )  On-demand  Self  Service:  On-Demand  Self  Service  allows  for  24-7  access  with 
little  to  no  human  interaction  once  an  SLA  between  the  customer  and  provider  is  in  place. 
The  ability  to  access  resources  "on  the  fly"  is  very  attractive  to  customers  and  huge 
benefit. 

2. )  Broad  Network  Access:  Cloud  has  ability  to  provide  the  network  access  and 
bandwidth  needed  to  accommodate  a  large  repository  of  data.  The  DoD  has  a  large  need 
for  access  to  Big  Data.  This  bandwidth  and  access  comes  at  a  cost,  but  with  the 
infrastructure  typically  already  in  place  (e.g.  Google's  fiber  expansion);  the  cost  is  at  a 
great  reduction  (Brooklyn,  2014). 
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3. )  Resource  Pooling:  The  ability  to  take  users  across  the  spectrum  and  spread  their 
use  over  all  the  resources  using  technologies  such  as  virtualization,  dynamic 
provisioning,  and  load  balancing  offers  a  distinct  benefit  to  operations  and  hardware 
budgets. 

4. )  Measured  Sendee:  This  is  the  ability  to  measure  and  quantify  the  user  utilization 
of  the  system.  This  characteristic  is  pertinent  to  DoD  cloud  utilization  and  worth  noting 
for  the  overall  view  of  its  history  and  background.  These  metrics  will  allow  DISA  to 
identify  improvement  areas. 

Cloud  Computing  Benefits 

Cloud  computing  offers  benefits  beyond  technology  used  by  the  DoD  to  date,  and 
implementation  should  be  viewed  by  the  Air  Force  as  paramount.  These  benefits  could 
overcome  the  challenges  of  access  to  Big  Data,  organizational  and  user  affects,  and 
acceptance  that  were  identified  in  Chapter  I.  In  addition  to  the  below  benefits,  this 
research  should  assist  in  overcoming  those  challenges.  Following  are  some  of  those 
benefits  offered  by  a  cloud  solution  (Barcomb,  2009): 

Continuous  Refresh:  Within  the  DoD  IT  realm,  refresh  schedules  are  critical  to 
ensuring  efficient  operation,  budget  and  manpower  forecasts.  The  current  process  puts  a 
huge  budgetary  constraint  on  managers,  further  hindering  upgradability  to  IT 
infrastructure.  Cloud  services  enable  smoother,  more  cost  efficient  refresh  capability  due 
to  its  inherent  virtualization  and  ability  to  transition  between  needed  platforms. 

Lower  Costs:  Overall  costs  in  the  DoD  for  IT  has  skyrocketed  in  recent  years.  Cloud 
offers  an  inherent  saving  in  its  ability  to  dynamically  control  users  "on-demand",  as  well 
as  the  ability  to  host  multiple  users  at  once.  There  are  spikes  in  utilization,  but  the 
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resources  are  redirected  depending  on  the  load,  creating  an  overall  reduction  in  cost  to  all 


users. 

Rapid  Elasticity:  This  characteristic  provides  reduced  costs  to  the  user  by  being  able 
add  or  take  way  hardware  and  software  resources  from  specifically  allocated  areas  of  use. 

Improved  Mission  Focus:  Decoupling  an  organization  from  its  data  and  the 
associated  applications  for  operations  reduces  the  IT  burden.  This  allows  the  organization 
to  increase  focus  on  the  mission  by  applying  resources  and  costs  that  were  previously 
associated  with  maintaining  elaborate  infrastructures,  and  data  and  application 
maintenance. 

Lower  Barriers  to  Entry:  Acquisition  actions  are  a  concern  for  DoD  due  to  limited 
availability  of  competent  contract  providers.  Cloud  solutions  give  a  window  of 
opportunity  to  smaller  contractors  offering  solutions  that  they  previously  could  not 
compete  with.  Harauz  et  al  describes  the  benefits  of  cloud  computing  to  small  businesses 
as  a  “major  selling  point”  (Harauz,  2009). 

DoD  Information  Assurance  (IA)  Guidelines 

Navigating  through  numerous  IT  guidelines  an  organization  can  take  advantage  of 
cloud  computing  benefits.  NIST  and  the  Federal  Information  Security  Management  Act 
offer  general  IT  governance;  however,  DoD  Directive  8500. 01E  "Information  Assurance 
(IA)  and  DoD  directive"  and  8500.2  "Information  Assurance  (IA)  Implementation"  get  to 
more  specifics.  All  of  these  documents  should  be  utilized  by  DISA  and  other  agencies 
when  looking  at  implementing  a  cloud  solution. 
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Business  Perspective 


Cloud  computing  has  changed  the  international  business  landscape  in  industry 
and,  with  this  research;  DISA  will  see  how  DoD  can  follow  suit  to  achieve  the  same 
success  experienced  within  the  civilian  sector.  There  are  many  business  factors  that  have 
been  successfully  enabled  by  cloud  computing.  These  factors,  which  can  transfer  to  the 
DoD,  justify  a  viable  model  development  resulting  from  this  research. 

Businesses  can  enhance  their  effectiveness  from  a  cloud  service  that  offers  such 
advantages  as  eliminating  barriers  to  entry,  providing  immediate  access,  lowering  IT 
costs,  enabling  enterprise  service  scalability,  and  pioneering  service  delivery.  Numerous 
notable  organizations,  such  as  Google,  IBM,  Microsoft,  and  AT&T,  have  taken 
advantage  of  these  benefits.  Their  successes  speak  for  themselves  (Marston  et  al,  2011). 
Understandably,  the  DoD  is  wary  of  implementing  new  technology,  but  this  research,  and 
the  track  record  of  successful  businesses  using  the  technology,  should  alleviate  those 
fears  and  provide  a  road  map  to  execution 

The  Government  Service  Agency,  who  is  moderating  the  cloud  brokerage,  has 
laid  out  possible  DoD  benefits  that  result  from  various  business  drivers  (Figure  3). 
Although  these  business  drivers  are  not  definitive,  they  can  be  compartmentalized  to  fit 
within  those  previously  mentioned. 
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Figure  3.  Business  Drivers  (Adapted  from  Cloud  Brokerage  Industry  Day,  2012) 


Implementation 

The  path  to  implementation,  if  decided  upon,  is  incumbent  on  DISA  and  will  be  an 
important  step  critical  to  the  success  or  failure  of  planned  completion.  The  following 
sections  will  further  assist  with  the  planned  implementation.  A  complete  implementation 
strategy,  however;  is  outside  the  scope  of  this  research.  A  partial  strategy  utilizing  this 
research  will  be  discussed  in  Chapter  V  while  answering  research  question  2. 
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Conclusion 


As  this  chapter  discussed,  cloud  computing  offers  many  advantages  and 
capabilities  that  should  be  taken  advantage  of  by  the  DoD.  Business  factors  that  are  tied 
to  a  successful  cloud  solution  and  implementation,  to  include  who  should  design  that 
process  and  the  flexibility  it  should  entail  were  identified.  As  Nicholas  (Carr,  2005) 
noted,  the  biggest  impediment  to  cloud  computing  “will  not  be  technological  but 
attitudinal”.  This  statement  is  true  when  looking  at  implementing  new  technologies  in  the 
DoD  and  cloud  computing  will  be  no  different.  This  analysis  should  offer  DISA  an 
effective  tool  to  ensure  implementation  success. 
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III.  Methodology 


The  Delphi  Technique  was  selected  as  the  methodology  of  this  research  paper  due 
to  its  ability  to  accurately  predict  future  characteristics  or  variables  that  may  affect  IT 
(Linstone  &  Turoff,  2002).  Schmidt  identifies  the  Delphi  method  as  a  valuable  tool  in  a 
researcher's  toolbox,  citing  "a  lack  of  a  definitive  method  for  conducting  the  research  and 
a  lack  of  statistical  support  for  the  conclusions  drawn."  The  Delphi  method  strengthens 
research  results  due  to  its  ability  to  represent  expert  opinions  instead  of  objective  facts 
(Dalkey  and  Helmer,  1962).  Cloud  providers  keep  market  research  and  business  models 
"close  to  the  vest",  which  makes  statistically  identifying  these  factors  difficult.  By  going 
to  the  experts  directly,  these  factors  can  be  identified  and  measured  based  on  how  they 
view  cloud  services  in  industry  and  DoD.  MoEs  for  the  business  factors  that  may  affect 
successful  implementation  will  be  quantified  using  the  Delphi  Technique.  Finally,  the 
Delphi  method  was  found  to  be  an  effective  means  in  regards  to  IT  studies  (Mishra  et  al, 
2002). 


The  Delphi  data  were  analyzed  using  Grounded  Theory  Methodology  (Charmaz, 
2007).  Grounded  Theory  Methodology  is  ideal  for  inductive  theory  building  because  it 
blends  the  best  of  qualitative  and  quantitative  methodologies.  Grounded  Theory  is 
generalizable  because  it  uses  systematic  sampling  procedures,  and  rigorous  because  it 
uses  systematic  coding  procedures  while  staying  "grounded"  in  the  subjects' 
interpretations  prior  to  enfolding  literature  (Strauss  and  Corbin,  1994).  Finally,  it  allows 
co-creation  between  the  researcher,  subjects,  and  literature  by  requiring  constant 
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comparison  between  the  three,  with  extensive  member  checking  (Glaser  &  Strauss,  1967; 
Glaser,  1978). 

In  this  thesis,  Grounded  Theory  was  applied  through  the  Delphi  method  in  three 
general  phases:  discovery,  analysis,  and  construction.  In  the  "discovery"  phase,  elicited 
texts  were  generated  in  the  form  of  the  Delphi  questionnaires.  It  was  this  phase  that  the 
Delphi  Methodology  and  Grounded  Theory  differed  the  most  because,  although  both 
methods  use  systematic  sampling  procedures,  the  Delphi  Methodology  seeks  experts  first, 
and  usually  does  not  expand  participation  beyond  initial  members,  whereas  Grounded 
Theory  is  known  for  adding  participants  (i.e.,  through  "snowball  sampling")  until 
theoretical  sufficiency  is  reached. 


Implementation 

The  following  explains  in  detail  how  the  Delphi  Technique  was  implemented  to 
achieve  satisfactory  results  from  a  panel  of  experts  specifically  selected  due  to  their 
experience,  knowledge  of  cloud  solutions,  and  status  as  managers,  developers,  and 
practitioners. 

The  study  started  by  selecting  the  panel  of  experts.  A  ‘first  pass’,  containing  a  set 
of  pre-determined  questions,  was  sent  to  the  experts,  allowing  them  to  respond  with 
open-ended  answers  (Appendix  D).  Upon  initial  sampling,  these  texts  were  "analyzed"  by 
coding  them  in  a  two-step  process  (open  coding  and  focused  coding)  that  allowed  the 
researchers  to  develop  shorthand  for  what  the  subjects  had  said,  first,  in  their  own  words, 
and  next,  using  words  and  phrases  shared  in  common  between  the  subjects.  This  step 
used  constant  comparison  between  subjects,  as  required  by  Grounded  Theory. 
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After  coding  the  responses,  the  process  of  theoretical  coding  was  used  to 
"construct"  diagrams  that  coalesced  all  subjects'  interpretations,  along  with  researcher 
inputs  (based  on  an  initial  literature  review  that  occurred  after  the  open  and  focused 
coding  processes).  The  construction  process  also  included  the  researcher  producing  brief 
memos  attempting  to  interpret  what  was  happening.  These  memos  were  the  genesis  of  the 
research  text  in  CH  4  and  5  of  this  thesis. 

A  summary  of  the  collected  responses  was  sent  to  the  panel  for  further 
clarification  regarding  their  view  on  the  validity  of  the  first  responses  and  comments 
were  requested  regarding  the  preliminary  models.  The  researchers  confirmed  subjects' 
interpretations  of  the  theoretical  codes— another  form  of  constant  comparison.  Upon 
incorporating  subjects'  ideas,  the  information  systems  and  organizational  literature  were 
again  consulted  to  compare  subjects'  interpretations,  researcher  inputs,  and  literature.  The 
responses  from  this  ‘second  pass’  (Appendix  D)  were  then  taken  and  consolidated  for  use 
in  the  development  of  final  models  containing  the  variables  identified  by  the  panel. 

In  the  third  Delphi  round,  the  resultant  theoretical  framework  was  once  again 
presented  to  Delphi  participants.  The  purpose  of  this  round  was  to  confirm  that 
convergence  was  reached,  satisfying  the  criteria  of  both  Delphi  and  Grounded  Theory  to 
reach  "theoretical  sufficiency"  (Charmaz,  2007).  This  last  round  of  inputs  served  to 
further  enhance  the  model  for  the  appropriateness  of  the  MoEs. 
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The  five  questions  asked  of  experts  in  the  ‘first  pass’  were  purposefully  vague  to 
elicit  an  in-depth  response  (Appendix  D).  This  took  advantage  of  the  expert's  experiences 
and  knowledge.  The  questions  were  as  follows: 

EXPERT  QUESTION  1:  What  factors  do  you  see  as  considering  successful  cloud 
computing  implementation  ? 

EXPERT  QUESTION  2:  What  are  your  key  concerns  for  successful  cloud  computing 
utilization  ? 

EXPERT  QUESTION  3:  What  are  the  main  obstacles  you  envision  hindering  successful 
cloud  computing  implementation  ? 

EXPERT  QUESTION  4:  What  do  you  see  as  differences  between  the  way  industry 
implements  a  cloud  solution  to  the  way  DoD  should? 

EXPERT  QUESTION  5:  How  would  you  overcome  those  obstacles  and  differences? 

An  analysis  was  performed  by  two  researchers  after  the  first  pass  responses  were 
received  using  Cohen's  Kappa  (Equation  1)  (Cohen,  1960).  The  recruitment  of  a  second 
researcher  was  pivotal  to  provide  a  different  perspective,  additional  rigor  to  the  research 
and  increase  validation  of  the  variables.  The  second  researcher  was  previously  trained 
and  intimately  familiar  with  the  grounded  theory,  the  coding  process,  and  the  research. 
The  output  from  the  coding  through  the  equation  below  is  identified  in  the  analysis 
portion  of  the  next  chapter. 

K  =  (O  —  E)  /(N  —  E) 

Where:  O  =Total  Agreement,  E  =  Total  rows/2,  the  division  representing  50%  agreement 
that  would  be  achieved  by  random  chance,  and  N  =  Toted  #  of  rows  in  the  analysis  of 
comments. 
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The  lower  limit  of  K  =  0  indicates  no  agreement  between  researchers,  0  -  0.20  as 
slight,  0.21  -  0.40  as  fair,  0.41  -  0.60  as  moderate,  0.61  -  0.80  as  substantial,  and  0.81  - 
1.0  as  almost  perfect  agreement.  The  upper  limit  of  K  =  +1.00  would  indicate  total 
agreement  between  researchers  (Cohen,  1960;  Landis  and  Koch,  1977). 

Knowledge  Areas 

Guidelines  have  been  established  as  to  how  to  determine  qualified  experts. 
Rigorous  steps  were  identified  by  Delbecq  et  al,  and  the  same  procedure  could  be  applied 
to  a  Delphi  study.  Since  a  statistical  sample  is  not  required,  a  qualified  panel  of  selected 
experts  is  sufficient.  The  Delphi  study  will  enable  model  accuracy  if  a  field  of  experts  is 
chosen  from  an  array  of  Knowledge  Areas  (KAs).  In  this  study,  the  three  KAs  are 
managers,  developers  and  practitioners.  Managers  will  provide  an  oversight  perspective 
to  how  cloud  environments  are  strategically  implemented  to  achieve  desired  results.  The 
developers  can  elaborate  as  to  how  a  cloud  environment  is  tactically  implemented  to 
achieve  the  directed  user  requirements.  Practitioners  are  on  the  leading  edge  of  cloud 
implementation  and  see  results  at  a  tactical  level  of  execution.  These  three  perspectives 
offered  representative  views  within  IT  that  provided  adequate  knowledgeable  responses 
and  resulted  in  sufficient  sampling  space  for  this  research. 

Using  established  protocols  (Figure  4),  experts  for  the  study  were  established  by 
preparing  the  Knowledge  Resource  Nominations  Worksheet  (KRNW)  and  setting  up  the 
panels  for  the  samples.  The  panels  were  classified  by  the  KAs. 
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Figure  4.  Procedure  for  selecting  experts  (adapted  from  Okoli,  2004) 

Managers 

This  KA  was  chosen  due  to  their  vast  experience  as  IT  managers.  The  oversight 
and  strategic  "big  picture"  that  they  offer  provides  the  linchpin  to  ensuring  IT  is 
effectively  utilized.  They  have  intimate  insight  into  what  works  and  what  does  not  within 
their  organizations. 

Developers 

This  KA  was  chosen  for  a  technological  view  relating  to  the  proper  creation  of  IT. 
How  a  technology  is  created  dictates  how  will  be  used.  They  possess  the  experience  and 
insight  into  the  implied  effects  of  the  technology  in  the  field  through  the  feedback  they 
receive  and  readjust  future  technologies  as  needed. 
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Practitioners 


This  KA  offers  a  "boots  on  the  ground"  view  of  how  the  technology  succeeds  or 
fails.  This  is  the  lowest  level  KA  in  this  study.  It  offers  a  critical  picture  of  what  will 
enable  a  cloud  computing  solution  to  ultimately  meet  the  needs  of  the  organization. 

Expert  Criteria 

No  set  method  for  declaring  an  individual  an  expert  has  been  established  for  the 
Delphi  method,  but  five  years  has  been  identified  as  a  viable  factor  (Mitchell,  1991; 
Rowe  and  Wright,  1999;  Dawson  &  Brucker,  2001).  The  experts  were  chosen  fulfill  this 
requirement  and  many  have  substantially  more  experience  beyond  their  knowledge  area. 
The  need  for  industry  representation  was  critical  to  this  research  to  enable  cloud 
implementation  and  is  important  to  the  DoD,  as  evidenced  by  the  following  statement: 

"The  DoD  Cloud  Computing  Strategy  has  been  expanded  to  address  use  of 
commercial  sendees  in  the  Department's  multi-provider  enterprise  cloud  environment. 
Adoption  and  implementation  of  commercially  provided  cloud  sendees  are  being  rapidly 
accelerated  with  the  maturing  of  the  Federcd  Cloud  Computing  Initiative,  the  Federcd 
Risk  and  Authorization  Management  Program  (FedRAMP),  and  release  of  the  2012 
National  Defense  Authorization  Act. "  (Teri  M.  Takai,  DoD  Chief  Information  Officer ) 

Demographics  of  the  experts  vary  (Table  1),  but  are  an  accurate  representation  of 
the  KAs  identified  earlier.  Varied  experiences  were  critical  in  formulating  a  viable  model 
and  lend  credibility  to  the  findings  of  this  study. 
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Table  1:  Expert  Demographics 


Duty  Area 

Knowledge 

Organization 

IT  Experience 

Cloud 

Operations 

Manager 

DISA 

17  yrs 

1  yr 

IT  Expert 

Practitioner 

ETTC  Corp 

Not  available 

Not 

Account 

Manager 

CSC 

5  yrs 

3  yrs 

Chief 

Manager 

CSC 

25  yrs 

13  yrs 

IT  Expert 

Practitioner 

AF  TENCAP/TCE 

15  yrs 

2.5  yrs 

Cyberspace 

Developer 

24  AF/A3X 

18  yrs 

5  yrs 

CEO 

Manager 

Craxel  Inc. 

Not  available 

Not 

IT  Expert 

Developer 

Next  Century 

Not  available 

Not 

Program 

Engineering 

Developer 

AFL/RCB 

2  yrs  /  30  yrs 
acquisitions 

2  yrs 

Research  Instruments 

The  instruments  in  this  study  are  used  to  perform  five  functions.  First,  was  the 
administration  of  the  Delphi  study  through  a  formal  request.  Then  the  KRNW  (Figure  4) 
process  was  used  to  select  the  experts  used  for  the  study.  Next  a  data  request  was 
deployed  to  acquire  answers  from  selected  experts  for  analysis.  The  fourth  instrument 
was  executed  to  analyze  that  data.  Finally,  model  creation  was  performed  to  fit  the 
research. 

Delphi  request 

In  accordance  with  the  Air  Force  Institute  of  Technology  policy  regarding  human 
subject  research,  an  Institutional  Review  Board  (IRB)  waiver  was  accomplished 
(Appendix  E).  The  waiver  was  approved  by  the  appropriate  reviewing  authority 
(Appendix  F).  This  approval  enabled  the  research  to  progress  in  a  timely  manner  and  was 
a  critical  step  in  this  research. 
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Subject  matter  expertise 

The  expert’s  subject  matter  expertise  level  assisted  in  ranking,  categorizing  and 
fitting  the  experts  in  the  correct  panel  of  samples,  without  consideration  of  their  names 
and  demographics.  This  process  of  classifying  experts  allowed  for  a  proper  sampling 
spread  across  all  the  KAs.  Some  experts  crossed  lines  of  expertise  based  on  their 
knowledge  and  the  positions  they  held  as  IT  professionals. 

Data  request 

The  data  request,  in  the  form  of  questions,  previously  identified  in  Chapter  III, 
were  sent  to  the  experts  via  questionnaire.  The  panel  was  instructed  to  answer  in  a  set 
time  for  responses  that  allowed  for  the  appropriate  amount  of  detail. 

Data  collection 

Data  collection  was  performed  by  the  researcher.  Responses  were  collected  and 
analyzed  using  independent  factor  analysis  using  Cohen's  Kappa  (Rosenthal  and  Rosnow, 
2007)  mentioned  earlier  in  this  section.  Data  were  then  grouped  together  to  formulate  the 
‘second  pass’,  a  response  back  to  the  experts  for  further  clarification  (Appendix  D).  The 
analysis  of  this  data  is  presented  in  Chapter  IV. 

Model  creation 

Model  creation  was  also  accomplished  by  the  researcher.  Once  the  responses  from 
the  experts  stabilized  and  a  consensus  reached,  the  models  were  formulated  accordingly. 
This  was  accomplished  within  the  three  to  five  rounds,  as  needed  (Linstone  &  Turoff, 
2002). 


26 


Conclusion 


This  chapter  narrated  the  specific  method  used  for  this  research.  It  expressed  how 
the  data  was  analyzed  and  how  grounded  theory  supported  that  analysis.  Next  it  showed 
how  the  techniques  were  implemented  in  a  pass  by  pass  description  of  the  Delphi  study. 
Also,  the  chapter  showed  the  selection  process  for  the  expert  panel  selection.  This  chapter 
then  explained  how  KAs  was  selected  for  categorization  of  the  experts.  Finally  the 
research  instruments  used  in  this  research  were  described.  The  Delphi  Technique  was 
chosen  due  to  its  effectiveness  in  IT  research;  this  chapter  conveyed  how  it  will  be  used. 
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IV.  Analysis,  Theory  and  Results 


Overview 

This  chapter  presents  the  expert  comment  analysis,  theory  that  supports  the 
research,  and  the  models  that  resulted.  The  analysis  will  show  the  output  from  the 
analysis  and  display  the  selected  variables  and  processes  that  fed  the  resultant  models. 
The  theory  will  show  the  technology  acceptance  theories  that  provided  the  foundation  for 
the  models.  Finally,  it  will  present  the  developed  models,  provide  detailed  explanation  of 
the  variables  and  processes  and  support  via  expert  comments. 

Analysis 

The  first  pass  of  the  survey  asked  respondents  to  answer  five  questions  associated 
with  the  impact  of  successful  cloud  computing  implementation  (Appendix  D).  The 
respondents  were  asked  to  be  specific  and  elaborate  as  much  as  they  deemed  necessary. 
The  questions  enabled  experts  to  be  as  flexible  as  needed  in  their  responses  to  ensure 
inputs  covered  the  vast  breadth  of  experience  and  knowledge  symbolic  of  the  expert 
demographic. 

The  analysis  from  the  first  passes  yielded  results  (Table  4)  using  Cohen's  Kappa 
(Rosenthal  and  Rosnow,  2007).  This  analysis  is  derived  by  researchers  identifying  main 
points  within  the  comments  from  the  experts  (Appendix  B).  Two  researchers  then 
compared  their  individual  results  to  extrapolate  the  variables  they  pulled  from  the 
comments. 

The  limits  of  agreement,  mentioned  in  Chapter  III,  are  the  basis  of  determining 
significance.  Given  the  limits  of  agreement,  the  K  =  .96  magnitude  factor  indicates  a 
match  to  the  extrapolated  comments  and  consensus  between  researchers  with  E 
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introducing  the  possibility  of  random  agreement  (Figure  5).  These  variables  (Table  2) 


were  sent  out  for  the  second  pass  (Appendix  D). 


K  =(0  -  E)  /  (N  -  E) 

N  =  100  0  =  98  E  =  50  (100/2) 

(98  -  50)  /  (100  -  50)  =.96  agreement 


Figure  5.  Researcher  Agreement 


The  first  responses  identified  similar  variables  from  both  the  DoD  and  industry 
experts.  These  variables,  derived  from  the  analysis  of  the  comments,  were  pulled  to  see 
how  many  times  the  experts  mentioned  those  areas  (Table  2)  to  indicate  relevance. 


Table  2:  Variable  Comments 


Variables 

#  of  Times 
Commented 

Understanding 

23 

Security 

7 

Budget 

9 

Access 

12 

Environment 

27 

Reliability 

7 

Standards 

17 

The  second  round  of  questions  requested  that  the  experts  rank  variables,  identified 
previously,  by  order  of  importance  with  1  indicating  most  important  (Table  3).  The 
second  pass  also  provided  evidence  that  the  preliminary  models  developed  were  viable 
when  trying  to  achieve  the  desired  goal  of  successful  cloud  computing.  Based  on 
responses,  both  preliminary  models  carried  a  consensus  amongst  the  experts.  The  process 
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model  had  the  highest  level  of  agreement  of  100%,  where  the  experts  agreed  with 


minimal  to  no  comments. 


Table  3:  Variable  Rank  Order 


Rank 

Variable 

#  of  Times  of 
Rank 

Rank  Total 

1 

Understanding 

1,1, 3, 1,3,1, 1,5 

16 

2 

Security 

2, 2, 5, 2, 4,3, 6, 2 

26 

3 

Budget 

5, 5, 4, 3, 1,2, 7, 3 

30 

4 

Access 

4,4, 1,4, 6, 7, 3, 4 

33 

5 

Environment 

3,7, 2,7, 7,6,2, 2 

36 

6 

Reliability 

7, 3, 6, 5, 2, 4, 5, 5 

37 

7 

Standards 

6, 7, 7, 6, 5, 5, 4, 4 

44 

Note:  lower  rank  total  reflects  the  most  important  factor 

Notice  the  rank  of  the  factors  (Table  3)  did  not  fit  the  amount  of  times  it  was 
commented  on  (Table  2).  There  was  nothing  noted  that  would  add  concrete  evidence  to 
this  effect  and  will  be  addressed  in  recommendations  for  further  research. 

In  the  third  and  final  pass,  finalized  models  were  sent  to  confirm  researcher’s 
findings  (Appendix  D).  The  preliminary  models  that  were  developed  in  the  second  pass 
were  further  enhanced  to  incorporate  expert  inputs.  Variables  used  in  the  variance  model 
were  ranked  by  the  experts  in  order  of  importance  (Table  3)  and  installed  into  the  model 
in  that  order.  The  process  model  needed  no  change,  as  there  was  consensus  on  that  model 
after  the  second  pass.  This  final  pass  resulted  in  consensus  from  all  experts  with 
comments  that  would  not  result  in  any  change  to  the  models  that  were  developed. 

Grounded  theory  requires  enfolding  the  literature  as  the  analysis  is  being 
performed.  As  this  analysis  was  conducted  it  became  evident  that  technology  acceptance 
theories  were  a  fit  for  the  research.  Those  supporting  theories  are  elaborated  in  the  next 
section. 
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Theory 


There  are  numerous  theories  that  exist  for  technology  acceptance.  Various 
theories  were  identified  for  use  in  designing  the  models.  All  theories  affect  how 
constructs  were  selected.  Some  constructs  were  a  better  fit  in  the  variance  model,  some 
were  a  better  fit  in  the  process  model,  and  still  others  fit  in  both. 

The  Unified  Theory  of  User  Acceptance  of  Technology  (UTUAT),  Technology 
Acceptance  Model  3  (TAM3)  and  Model  of  Technology  Acceptance  by  Groups  (M- 
TAG)  affect  which  model  the  constructs  were  placed  in  and  are  the  best  theories  that 
supported  this  research.  The  variables  in  the  variance  model  were  formed  after  close 
analysis  of  expert  comments  by  researchers.  These  variables  include  understanding, 
security,  budget,  access,  reliability,  environment,  and  standards.  The  process  model 
identified  comments  from  experts  as  processes  and  then  separated  into  levels  identified  as 
DoD,  organizational,  technology,  and  user.  This  separation  was  accomplished  to 
delineate  where  each  level  exists  and  analysis  of  this  model  will  follow  accordingly. 

Pre-User  Acceptance 

The  variables  and  processes  in  both  models  exist  in  a  variety  of  constructs 
regarding  technology  acceptance  in  examinations  by  Bailey  and  Pearson  (1983),  Baroudi 
and  Orlikowski  (1988),  Doll  and  Torkzadeh  (1988),  and  Ives  et  al.  (1983),  and  are 
represented  in  Appendix  A  derived  from  Wixom  and  Todd  (2005).  These  constructs  were 
formed  prior  to  most  user  acceptance  research  streams  {TAM,  TAM2,  UTAUT,  TAM 3) 
and  in  turn  led  to  the  elaboration  of  theories  linked  to  this  research  {UTAUT,  TAM3,  M- 
TAG). 
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User  Acceptance  Theories 


User  acceptance  focuses  on  analysis  of  new  technologies,  such  as  cloud 
computing,  should  be  accomplished  to  enable  deployment  by  managers.  User  acceptance 
understanding  can  enhance  the  use  of  appropriate  models  to  support  that  deployment, 
implement  proper  procedures,  and  promote  efficient  communications  (Kendall,  1997). 
There  are  numerous  theories  that  exist  for  technology  acceptance  and  will  serve  to  aid 
that  necessary  analysis. 

UTAUT,  TAM3  and  M-TAG  were  found  to  be  the  best  fit  to  the  results  found 
during  the  research.  TAM3  was  chosen  over  TAM  (Figure  6)  and  TAM2,  because  TAM 
results  have  been  shown  to  be  unclear  and  TAM2  does  not  represent  a  broad  enough 
scope  needed  for  this  research  (Legris  et  al,  2003). 

A  model  for  SaaS  adoption  by  Wei-Wen  Wu  posited  in  the  2011  article, 
"Developing  an  Explorative  Model  for  SaaS  Adoption",  is  noteworthy  and  will  be 
discussed  later  in  this  chapter.  IT  research  has  expanded  into  recent  research  concerning 
technology  acceptance  by  groups,  notably  an  article  by  Saonee  Sarker  and  Joseph 
Valacich  titled  "A  Non-Reductionists  Approach  to  Studying  Technology  Adoption  by 
Groups"  which  draws  on  previous  theories  into  technology  acceptance  at  the  group  level 
versus  prior  user  theories.  While  all  of  these  theories  were  used  to  structure  the  two 
models  in  this  research,  some  were  more  critical  than  others  and  will  require  a  more  in- 
depth  understanding. 
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Figure  6.  TAM  (Adapted  from  Venkatesh  and  Davis,  2000) 


User  Acceptance  History 

TAM  was  developed  by  Fred  Davis  as  one  of  the  first  IT  user  acceptance  models 

formulated  and  accepted  in  academic  literature.  In  his  research,  he  articulated  ‘Perceived 

Usefulness’  and  ‘Perceived  Ease-of-Use’,  both  defined  below  (Davis,  1989).  These  two 

concepts  were  the  basis  of  TAM  and  succeeding  models  since  have  provided  the 

connection  of  behavior  to  other  predictors  of  usage  in  that  model.  TAM,  although 

providing  predictability,  was  still  limited  by  its  influence  due  to  its  inability  to  provide 

feedback  regarding  value  (Taylor  and  Todd  1995,  Venkatesh  et  al.  2003).  That  user 

feedback  value  would  be  important  for  leadership  buy-in  and  effective  implementation. 

Perceived  Usefulness  (PU):  The  degree  to  which  a  person  believes  that  using  a 
particular  system  would  enhance  his  or  her  job  performance. 

Perceived  Ease-of-Use  (PEOU):  The  degree  to  which  a  person  believes  that  using 
a  particular  system  would  be  free  from  effort. 
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Further,  there  was  support  identifying  the  relationship  between  technology 
implementation  when  associated  with  adoption  and  continued  usage.  This  was  evident 
after  a  study  of  Windows  3.1  implementation  (Karahanna  et  al,  1999).  The  differences 
between  usage  and  adoption  in  the  Windows  3.1  research  was  taken  into  account  when 
determining  the  factors  and  processes  in  the  two  models  developed  within  this  research.  It 
would  seem  that  non-mandatory  usage  initially  will  develop  successful  cloud 
implementation. 

An  additional  concept  that  contributed  to  creating  an  acceptance  construct  was 
TAM2,  an  expansion  of  TAM.  Both  theories  have  pushed  technology  acceptance 
research  further;  however,  TAM2  (Figure  7)  adds  three  additional  variables  of  subjective 
norm,  voluntariness,  and  image  (Venkatesh  and  Davis,  2000).  Although  a  direct 
relationship  between  TAM2  and  the  developed  research  models  is  not  apparent, 
understanding  the  theory  is  germane  to  the  research.  As  Pfeffer  (1982)  postulates,  an 
individual  "achieves  membership  and  the  social  support  that  such  membership  affords  as 
well  as  possible  goal  attainment  which  can  occur  only  through  group  action  or  group 
membership.  ” 


34 


Figure  7.  TAM2  (Adapted  from  Venkatesh  and  Davis,  2000) 


Opitz  et  al.  (2012)  ties  cloud  computing  acceptance  directly  to  TAM  models  via 
his  research  article,  "Technology  Acceptance  of  Cloud  Computing  Empirical  Evidence 
from  German  IT  Department.  ”  He  concludes  that  empirical  research  was  offered  to  link 
technology  acceptance  models  to  the  cloud  computing  technology  that  this  research  is 
directed  towards. 

Theories  Linked  to  this  Research 

TAM  and  TAM2  have  direct  tie-ins  to  this  research.  However,  UTAUT,  TAM3 
and  M-TAG  had  a  more  influence  on  the  formation  of  the  two  models  that  were 
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developed.  By  following  the  matrix  in  Appendix  A  one  can  see  the  association  between 
these  theories  and  how  they  exist  within  the  models  is  evident. 

UTAUT  (Figure  8)  is  a  recent  acceptance  theory  that  was  essential  to  the 
production  of  the  two  models  presented.  Part  of  this  theory  reinforces  the  idea  of  "job 
outcome  of  interest"  as  a  determinant  in  successful  technology  implementation  building 
upon  the  key  UTAUT  constructs  of  performance  expectancy,  effort  expectancy,  social 
influence  and  facilitating  conditions.  (Venkatesh,  2003).  The  construct  of  job  outcome 
was  presented  within  a  synthesis  of  other  technology  acceptance  theories  in  "Dead  or 
Alive?  The  Development,  Trajectory  and  Future  of  Technology  Adoption  Research" 
(Venkatesh  et  al,  2003)  and  was  a  key  factor  when  analyzing  the  various  factors  affect  the 
models.  One  of  the  focuses  of  UAUT  encompasses  operational  beliefs  and  the  ability  to 
perform  ones  job  via  PU  and  PEOU.  This  will  be  addressed  in  the  model  explanation  in 
various  constructs. 


Figure  8.  UTAUT  (Adapted  from  Venkatesh  et  al,  2003) 
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TAM3  (Figure  9)  was  developed  by  combining  TAM2  and  the  components  of  the 
model  that  lead  to  perceived  ease  of  use  (Venkatesh,  2000).  It  was  suggested  and  found 
in  Venkatesh's  "Technology  Acceptance  Model  3  and  a  Research  Agenda  on 
Interventions , "  that  the  components  that  lead  to  perceived  ease  of  use  do  not  influence 
perceived  usefulness.  This  delineation  between  the  two  theories  is  critical  to  ensuring  that 
TAM3  can  present  a  separate  research  stream  within  the  community  (Venkatesh  and 
Bala,  2008).  The  linkage  between  user  acceptance  and  group  theories  lies  in  categorizing 
the  items  in  the  models. 

TAM3  was  employed  to  formulate  the  main  variables  in  the  variance  model  that 
is  presented  in  the  results  section,  although  after  elaboration  from  the  experts,  UTAUT 
and  M-TAG  further  supported  that  development.  It  was  found  in  Venkatash  and  Bala 
(2008)  that  the  constructs  of  understanding,  access,  reliability,  environment,  and 
standards  led  to  acceptance.  These  ideas  are  found  in  the  variance  model.  Expert 
comments  showed  that  those  same  constructs  exist  at  the  group  level.  The  construct  of 
budget  could  not  be  found  in  any  theories.  Based  on  expert  comments  on  this  research 
(Appendix  C),  it  can  be  concluded  that  lack  of  adequate  budget  could  be  DoD  specific. 
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Figure  9.  TAM3  (Adapted  from  Venkatesh  and  Bala,  2008) 


M-TAG  was  developed  recently  and  has  been  analyzed  in  articles  about 
Technology  Adoption  by  Groups  (TAG)  such  as  "Technology  Adoption  by  groups" 
(Sarker  et  el.,  2005)  as  well  as  the  direct  production  of  M-TAG  in  "An  Alternative  to 
Methodological  Individualism:  A  Non-Reductionist  Approach  to  Studying  Technology 
Adoption  by  Groups"  (Sarker  and  Valacich,  2010)  (Figure  10).  M-TAG  was  a  key  theory 
that  fits  the  process  model  that  is  elaborated  in  a  later  section.  The  individual  processes 
within  the  process  model  levels  of  DoD,  Organizational,  User  and  Technology  also  exist 
within  most  technology  and  user  acceptance  theories  (Table  2  and  Table  3). 

M-TAG  also  added  evidence  to  the  validity  of  the  variables  in  the  variance  model. 
The  constructs  of  understanding,  access,  reliability,  environment,  and  standards  are  all 
user  acceptance  constructs  and  were  found  to  have  a  high  correlation  to  a  group's  strength 
of  adoption  of  technology  (Sarker  and  Valacich,  2010).  Once  again,  security  and  budget 
are  not  key  constructs  in  the  theory,  but  were  emphasized  by  experts  in  their  comments 
during  this  research. 
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Using  the  same  constructs  as  current  technology  acceptance  models,  there  has 
been  an  identified  association  with  MIS  (Management  of  Information  Systems)  success 
in  Electronic  Data  Processing  systems  in  small  businesses  which  could  affect  larger 
organizations  as  well  (Delone,  1988;  Raymond,  1985).  Previous  research  has  shown 
relationships  between  IT  business  value  and  organizational  performance,  depending  on 
business  values  such  as  management  practices,  structure,  and  environment  (Brynjolfsson 
et  al,  2002;  Cooper  et  al,  2000;  Dewan  and  Kraemer,  2000).  This  is  reflected  in  this 
research  from  the  perspective  of  mission  success  and  budgetary  constraints  as  the 
business  values  that  exists  in  the  DoD. 


Figure  10.  M-TAG  (Adapted  from  Sarker  and  Valacich,  2010) 


39 


Finally,  SaaS  model  adoption  should  be  mentioned  since  it  is  the  cloud  broker 
service  that  DISA  plans  to  implement.  Few  studies  have  been  accomplished  regarding 
SaaS,  and  as  a  cloud  model,  SaaS  adoption  research  should  be  analyzed  in  this  section. 
The  association  of  technology  adoption  constructs  associated  with  marketing  efforts, 
security  and  trust  leading  to  SaaS  adoption,  set  a  good  foundation  for  this  research  (Wu, 
2011).  A  more  in-depth  study  in  the  relations  between  industry  and  DoD  is  necessary. 
This  research  accomplished  that  with  a  good  cross  sectional  selection  of  experts  and  is 
presented  in  the  next  section. 


Results 

The  answers  from  the  experts  in  the  three  step  methodology  of  the  research 
resulted  in  two  viable  models.  The  first  model,  referred  to  as  the  variance  model, 
expresses  variables  that  could  affect  the  result  of  successful  cloud  computing 
implementation.  The  second  model,  identified  as  the  process  model,  communicates 
processes  that  could  affect  the  execution  of  the  cloud.  Both  of  these  models  answer 
research  question  3,  "Can  a  ''model'1  be  developed  that  will  assist  DISA's  strategy  for 
successfully  fielding  cloud  computing  in  the  DoD?”  The  variables  and  processes  within 
the  two  models  answer  research  question  1,  "What  organizational  variables  or  processes 
influence  successful  cloud  computing  implementation  in  the  DoD?"  These  variables  and 
processes  are  also  listed  in  Table  4.  Research  question  2  will  be  discussed  in  Chapter  V. 
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Table  4:  Variables  and  Processes 


Variables 

Understanding 

Security 

Budget 

Access 

Reliability 

Environment 

Standards 

Processes 

Security  enclaves 

Existing  broker  reputation 

Combat  support 

Existing  IT  staff  maturity 

Security  policy 

Existing  policy  &  governance 

Contracting  process  flexibility 

Entrepreneurial  leadership/vision 

Sense  of  urgency 

Education 

Service  advocacy 

Advertising 

Old  Think 

Marketing 

Change  averse  culture 

Put  the  "right  stuff"  in  the  cloud 

Certification/ Accreditation  process 

Reduce  barriers  to  entry/bureaucracy 

Non-monetary  value 

Development  enclaves 

Situational  awareness 

Holistic  implementation 

Clearly  defined  requirements 

T  ime/timing/phasing/mi  gration 

Flexible  configurations 

Evolution  roadmap 

Data  integrity 

Clear  objectives 

Security 

Incentives 

Near-zero  latency 

Organization  support 

Reliability 

Mature  business  model 

Interoperability 

Flexible  options 

Access  to  service 

Cost  savings 

Support  to  users 

Productivity  improvement 

User  expectations/understanding 

Trust/support  for  initiative 

Models 


The  previous  sections  set  the  foundation,  based  on  literature,  of  the  various 
theories  that  support  these  models,  but  the  "golden  thread"  to  this  research  lies  in 
UTAUT,  TAM3  and  M-TAG.  Also  to  note,  constructs  from  the  previously  discussed 
theories  are  all  contained  in  the  two  models  directly  or  indirectly.  The  variance  model 
consists  of  seven  constructs  and  one  moderating  variable  that  were  categorized  based  on 
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expert  comments  and  identified  from  the  preceding  technology  and  group  acceptance 
theories.  The  process  model  contained  four  levels  (DoD,  organizational,  technology,  and 
user);  within  those  levels  were  numerous  processes  found  to  be  key  in  data  collection  in 
relation  to  this  type  of  model. 


Variance  Model 

A  variance  model  is  a  ‘cause  and  effect’  model.  It  provides  an  explanation  of  how 
input  variables  cause  an  effect  on  a  dependent  variable  (Successful  Cloud  Computing 
Implementation  is  the  dependent  variable  in  the  model).  These  factors  can  have  a  positive 
or  negative  effect  on  that  variable,  as  any  number  of  additional  factors  could  have  an 
effect  on  the  output  variable.  This  type  of  model  can  be  translated  into  a  mathematical 
equation. 

Y  =  mlX  +  m2Y  +  m3XY . +  b  +  e 

Where  m  =  slope,  X,  Y  =  variables ,  XY  =  intersection  between  two  variables  in  a 
multivariate  model ,  b  =  intercept  and  e  =  error. 

The  results  from  the  study  generated  a  variance  model  (Figure  11)  with  factors 
that  answer  research  question  1.  This  model  shows  the  seven  factors  identified  (Table  3) 
and  one  moderating  variable  that  was  shown  to  affect  one  of  the  factors.  Factor  support 
was  derived  from  analyzing  comments  from  experts  and  was  used  in  the  factor 
breakdown  earlier  described  (Appendix  B).  The  comments  explain  the  affects  displayed 
in  the  variance  model. 
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Figure  11.  Variance  Model 


The  variance  model  contained  the  constructs  of  understanding,  security,  budget, 
access,  reliability,  standards,  and  environment,  with  a  moderating  variable  of  training 
directly  impacting  understanding.  UTAUT,  TAM3,  and  M-TAG  all  that  were  based  off 
TAM  contributed  to  the  creation  of  this  model.  The  following  few  sections  describe  how 
those  theories  tie  to  this  model  and  how  they  affect  successful  implantation. 
Understanding 

The  experts  agreed  that  understanding  was  the  most  important  variable  that 
predicts  successful  cloud  implementation.  Their  definition  of  understanding  centered  on 
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users  knowing  what  cloud  computing  is,  what  it  provides  (job  relevance),  what  they  want 

from  the  cloud  (result  demonstrability),  and  what  constitutes  success  (output  quality) 

leading  to  PU  and  PEOU  within  TAM2  (Venkatesh  and  Davis,  2000). 

Common  understanding.  There  are  many  terms  being  used  to  describe  what  "cloud'' 
is/isn 't  which  has  caused  a  significant  amount  of  confusion.  A  significant  portion  of  the 
populace  only  thinks  of  the  “cloud"  from  Apple,  Google,  Amazon  or  IBM  commercial 
offerings  to  store  and  access  data.  They  don ’t  understand  that  there  are  utility  and 
analytic  clouds  as  well.  The  picture  gets  muddier  when  other  terms  like  "net-centric, " 
"SO A, "  "IaaS, "  "PaaS, "  "SaaS, "  etc.  are  discussed.  (Expert  4  Response) 

The  literature  supports  this  because  within  all  theories  of  TAM  a  user  must  first 
understand  what  a  technology  is  and  what  is  supposed  to  do  before  they  can  formulate  a 
perception  of  its  usefulness.  In  all  TAM  literature  perception  of  usefulness  is  a  strong 
determinant  to  behavioral  intention  to  use  a  technology  (Venkatesh  et  al,  2003).  This 
finding  is  also  supported  by  task-technology  fit  literature  and  TAM3  which  states  that 
users  perception  of  the  fit  between  the  technology  and  the  tasks  could  result  in  successful 
usage  (Ahuja  and  Thatcher,  2005;  Vanketash  and  Bala,  2008).  These  two  theories  imply  a 
third  idea  which  is  critical  in  a  cloud  context:  training. 

Training  was  not  specifically  identified  as  one  of  the  main  variables.  It  was  added 
as  a  moderating  variable  after  various  comments  addressed  the  need  for  the  user  to 
understand  and  use  cloud  computing  and  was  concurred  by  experts  in  later  Delphi  passes. 
A  moderating  variable  is  a  variable  that  can  affect  the  strength  of  the  relationship 
between  independent  and  dependent  variables.  (Cohen  and  Cohen,  1975).  In  TAM 
literature,  training  is  an  essential  component  of  a  user  formulating  perceptions  about  the 
use  of  a  technology.  Also,  training  was  found  in  the  pre-TAM  theories  as  confidence  in 
the  system  and  degree  of  training  (Bailey  and  Pearson,  1983).  Next,  training  support  was 
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found  in  TAM2  through  result  demonstrability  as  well  as  both  TAM2  and  TAM3  through 
output  quality  (Venkatesh  and  Bala,  2008).  Finally  training  is  evident  in  M-TAG  where 
"strong  training  mechanisms  initiated  to  reduce  the  group ’s  perceived  complexity  of  the 
technology  may  help  elevate  the  valence  towards  the  technology"  (Sarker  and  Valacich, 
2010).  Since  cloud  is  new  to  DoD  and  is  intangible,  training  will  be  critical  to  user 
understanding 
Access 

Our  experts  also  identified  access  to  the  technology  as  an  important  facilitating 

condition.  They  defined  access  as  the  ability  to  connect  to  the  system  with  minimal 

barriers.  This  is  a  training  issue  as  well  as  a  technical  issue  and  is  addressed  in  the  TAM 

literature  with  the  construct  ease  of  use  (Davis,  1989;  Venkatesh  and  Davis,  2000).  Over 

time,  if  access  is  difficult  and  hard  to  acquire  then  the  user  will  stop  using  the  system. 

Environment  should  be  vertically  integrated  to  provide  seamless  access  at  both  a 
Infrastructure  as  a  Sendee  level  and  a  Platform  as  a  Sendee  level.  Bad  choices  of 
mismatched  products  at  each  level  of  the  stack  will  lead  to  poor  adoption.  Resource 
management  must  be  built  into  the  core  of  the  environment  to  allow  for  the  seamless, 
automated  allocation  of  resources  based  on  policy  and  not  manual  allocation  based  on 
trouble  ticket  like  requests  to  a  help  desk.  ( Expert  3  Response) 

Reliability 

This  same  argument  holds  for  the  reliability  of  the  system  which  our  experts 
defined  as  a  performance  issue.  If  the  system  is  not  reliable  over  time  the  user  will  cease 
to  use  it.  This  is  addressed  in  the  TAM  literature  as  perceived  ease  of  use  (Davis,  1989; 
Venkatesh  and  Davis,  2000). 

Reliability:  Solution  needs  to  be  up  all  the  time  (Expert  5  Response ) 
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Security 

The  experts  identified  security  as  the  second  most  important  variable  leading  to 
successful  cloud  implementation.  They  defined  security  as  the  ability  to  trust  that  the 
system  will  protect  the  information  with  minimal  risk.  In  the  TAM  literature  security 
spans  three  categories:  usefulness  (TAM/TAM2),  ease  of  use  (TAM/TAM2)  (Venkatesh 
and  Davis,  2000),  and  facilitating  conditions  within  UTAUT  (Venkatesh  et  al,  2003). 
Since  security  is  a  critical  tool  (useful)  and  a  feature  (usable),  if  faith  is  lost  then  usage 
could  cease  further.  If  data  are  breached  or  system  is  disabled,  usage  will  not  be  possible. 
Hence  the  elevated  importance  in  a  DoD  context. 

Security.  With  improved  access  and  availability  of  data/information,  the  importance  of 
maintaining  the  security  and  integrity  of  the  data  becomes  paramount.  (Expert  4 
Response) 

Standards 

Integrally  related  to  security  as  well  as  usability  features  are  standards,  since 
standards  impact  both.  Our  experts  define  standards  as  policies,  processes,  and  guarantees 
with  respect  to  services  and  security.  TAM  literature  (UTAUT)  refers  to  this  as 
facilitating  conditions  (Venkatesh  et  al,  2003).  Standards  both  cause  and  are  a  product  of 
the  environment  and  have  an  impact  on  usage. 

Policy.  One  of  the  most  significant  challenges  will  be  to  develop  the  information  sharing 
mindset  and  implement  powerful  guidance/standards  to  force  the  issue.  Despite 
numerous  senior  leader  decrees,  directives  and  orders,  organizations  still  think  they 
“own  ”  the  data.  ( Expert  4  Response) 

Environment 

While  they  showed  difficulty  converging  on  one  definition  of  environment  they 
all  agreed  on  the  effect  that  it  had.  They  agreed  that  it  either  facilitated  or  hindered 
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successful  cloud  computing  implementation.  Further,  they  agreed  that  the  DoD 
environment  was  problematic.  TAM  literature  (UTAUT)  defines  environment  as  a 
facilitating  condition  that  directly  affects  usage  regardless  of  user  perception  (Venkatesh 
et  al,  2003).  Environment  was  also  found  to  be  a  key  variable  when  studying  group 
relationships  to  IT  structures  and  provides  a  positive  effect  on  successful  implementation 
(Reinig  and  Shin,  2002).  Environment,  as  evident  in  literature  and  supported  by  expert 
comments,  is  critical  to  IT  implementation.  DISA  should  be  aware  of  its  importance 
throughout  the  implementation  process  of  cloud  computing. 

If  the  service  provided  is  valuable  to  the  customers  using  it,  the  environment  will  deliver 
value  through  higher  rates  of  usage,  productivity  improvements  and  other  “soft” 
benefits.  ( Expert  7  Response) 

Budget 

The  experts  agreed  that  budget  was  very  important  (#3  on  Table  3).  They  defined 
budget  as  the  money  spent  on  a  system,  the  time  it  took  to  implement,  and  the  results  in 
terms  of  return  on  investment.  It  seems  that  TAM  (UTAUT)  addresses  this  indirectly  as 
a  facilitating  condition,  for  example  long  lead  times  to  a  budget  could  kill 
implementation.  It  seems  reasonable  that  DISA  needs  to  look  at  the  preceding  factors  to 
determine  a  reliable  ROI.  If  this  return  is  not  adequate,  then  cloud  computing 
implementation  will  flounder  as  a  business  model. 

The  government  budget  cycle:  It  works  for  acquisition  programs  that  span  many  years, 
but  it's  not  well  suited  to  ill  or  undefinable  requirements  that  are  subject  to  change 
rapidly.  To  steal  a  quote  from  my  MILCOM  paper  “ DoD  acquisition  managers  are  under 
constant  pressure  to  maintain  currency  across  their  enterprises  and  meet  ever  changing 
requirements  over  an  increasingly  complex  infrastructure.  ( Expert  3  Response) 
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Upon  completion  of  the  analysis  of  the  variance  models  it  is  apparent  that  the 
expert  opinions  are  well  supported  by  the  technology  acceptance  theories.  All  constructs 
in  the  variance  model  were  supported  by  one  or  more  user  acceptance  theories 
(Appendix  A).  However,  it  was  also  apparent  that  the  variance  model  was  not  fully 
explanatory,  because  it  did  not  take  into  account  the  order  in  which  events  occurred,  nor 
the  levels  of  the  organization  in  which  they  occurred.  In  order  to  fully  capture  the 
richness  of  the  experts’  comments,  another  model  was  necessary.  While  a  variance 
model  is  good  at  explaining  ‘what’  and  ‘why’,  a  process  model  is  useful  in  explaining 
‘how’  and  ‘when’.  Hence,  the  next  step  of  analysis  required  construction  of  an 
explanatory  process  model  to  fully  capture  expert  responses  and  recommendations. 

Process  Model 

Where  a  variance  model  is  good  at  explaining  ‘what’  and  ‘why’,  a  process  model 
is  tailored  to  explain  ‘how’  and  ‘when’.  A  process  model  provides  a  rational  explanation 
of  the  processes  by  evaluating  possible  courses,  or  paths,  based  on  observed  factors.  It 
provides  "linkage"  between  those  factors,  where  each  standard  needs  to  be  fulfilled 
before  moving  to  the  next  until  the  model  reaches  completion.  Often,  a  process  model 
may  contain  feedback  loops  between  any  identified  factors  within  the  model. 

The  second  model  generated  from  the  results  of  the  study  was  a  process  model 
(Figure  12).  This  model  shows  the  processes  and  how  they  interact  within  the  model,  as 
well  as  between  differing  levels.  The  processes  identified  also  aid  in  answering  research 
question  1.  Process  support  was  derived  from  comments  from  experts  and  used  in  the 
analysis  earlier  described  (Appendix  C). 
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DoD  Specific  Challenges 
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Figure  12.  Process  Model 


levels 


In  order  to  simplify 
containing  multiple 


the  analysis,  the  process  model  was  broken  down  into  four 
processes  within  and  between  each  level.  For  ease  of 
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explanation,  this  thesis  will  discuss  this  model  level  by  level,  highlighting  the  main 
processes  and  how  they  tie  into  one  another  culminating  in  successful  cloud  computing 
implementation.  In  some  cases  processes  outside  the  areas  of  analysis  were  aggregated. 
This  was  accomplished  for  ease  of  explanation  regarding  theory  support. 

DoD  Level 

Within  the  DoD  level,  the  experts  defined  two  challenges  of  successful  could 
computing  implementation  that  both  seem  to  be  uniquely  related  to  the  DoD.  Both  of 
these  challenges  produced  five  effects  that  trickled  downstream  in  the  model  (Figure  13). 


Figure  13.  Process  Model  DoD  Level 


The  top  level  of  DoD  specific  challenges  contains  the  use  cases  unique  to  the 
military  environment.  This  process  was  not  specifically  mentioned  in  technology 
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acceptance  theories,  but  is  germane  to  this  research  since  it  was  qualified  by  the  expert 
panel  as  important  in  relation  to  security  and  support.  The  experts  defined  this  as  multi¬ 
level  security,  combat  support,  DoD  acquisition  process,  and  contracting  flexibility. 
Within  this  level,  however,  fear  of  the  unknown,  acquisition  process,  and  DoD  culture 
can  cross  reference  to  technology  acceptance  of  groups  and  organizational  success 
through  variables  of  complexity,  uncertainty,  and  management  culture.  These  factors 
were  found  to  affect  technology  acceptance  in  groups  (Raymond,  1990). 

In  order  to  explain  the  effect  that  DoD  unique  cases  have  on  cloud  technology,  we 
have  to  understand  the  purported  benefits  in  industry  as  compared  to  DoD.  In  industry  the 
benefits  is  turning  infrastructure  into  service,  but  the  only  way  to  do  this  is  to  buy  a 
commercial  solution.  Also  ease  of  access,  commercial  availability  and  standard  solutions 
are  key  benefits  of  a  commercial  cloud.  The  experts  agreed  that  DoD  might  negate  these 
benefits  through  unique  cases  by  attempting  to  modify  or  recreate  commercially  available 
solutions. 

The  DoD  use  cases  and  implementation  are  different  than  the  Industry  use  cases.  (Expert 
6  Response ) 

Our  experts  defined  the  DoD  culture  with  respect  to  cloud  implementation  as 
consisting  of  service  advocacy,  fear  of  change  and  ominous  regulations  collectively 
termed  "old  think".  Taken  both  individually  and  as  a  whole,  they  anticipated  that  this 
would  negatively  affect  the  cloud  implementation  environment.  Shein,  1985  expressed 
culture  in  three  levels  consisting  of  behaviours  and  artefacts,  values,  and  basic 
assumptions.  Further  elaboration  of  these  levels  is  listed  below. 
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DoD  has  a  culture/people/institutional  barrier  to  think  and  behaving  as  a  business 
enterprise  in  areas  where  that  think  could  benefit  them  the  most.  If  you  want  to  be  an 
enterprise,  think  and  behave  as  one.  (Expert  7  Response) 

Shein  as  stated  by  Lim  (1995),  essentially  argues  that  culture  can  shape  an 
internal  environment,  and  in  doing  so  will  affect  organizational  behavior.  So  if  the 
primary  cultural  drivers  in  DoD  are  negative,  then  the  environment  will  be  unstable  or 
infertile  with  respect  to  cloud  implementation.  Lim's  research  also  argues  that  the  effects 
of  culture  extend  to  the  IT  environment. 

A  stable  organizational  environment  affects  technology  quality  by  enabling  the 
ability  to  implement  capable  technologies  in  an  IT  friendly  atmosphere  (Raymond,  1985). 
This  type  of  environment  could  lead  to  greater  ingenuity  in  IT  development  resulting  in 
better  quality  technology.  The  model  also  showed  an  effect  between  environment  and 
productivity.  This  association  is  cemented  in  the  works  of  Marcoulides  and  Heck  (1993) 
where  a  direct  correlation  was  shown  between  a  friendly  organizational  environment 
(climate)  leading  to  increased  productivity  (performance). 

My  main  concerns  are  lack  of  a  ‘good’  customer  base  both  for  implementation  and 
utilization,  organizational  churn,  and  a  weak  support  model.  (Expert  6  Response) 

From  this  we  can  speculate  that  the  organizational  environment  with  respect  to 
cloud  implementation  will  impact  the  quality  of  the  technology  and  its  implementation 
and  how  productive  the  organization  will  be  as  a  result  of  its  implementation.  Finally,  the 
Technological  Imperative  Model  developed  by  Orlikowski  (1992)  emphasizes  that 
technology  has  a  direct  relation  to  organizational  dimensions  such  as  individual 
communication  effectiveness,  skill  levels,  job  satisfaction,  and  productivity. 
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However,  if  the  sendee  provided  is  valuable  to  the  customers  using  it,  the  environment 
will  deliver  value  through  higher  rates  of  usage,  productivity  improvements  and  other 
“soft”  benefits.  (Expert  7 Response) 

This  section  argued  that  these  constructs,  if  addressed,  pre-implementation,  will 
increase  the  likelihood  of  successful  cloud  computing  implementation.  Realistically, 
these  are  long  lead  times  that  we  cannot  change,  so  in  a  DoD  implementation  you  would 
have  to  overcome  those  obstacles. 

Organizational  Level 

The  experts  identified  numerous  processes  within  the  organizational  level.  Those 
processes  were  broker  reputation,  IT  maturity,  leadership  vision,  training  culture,  policy, 
flexible  configuration,  environment,  and  economic  factors.  These  processes  produced 
three  effects  downstream  (Figure  14). 
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Figure  14.  Process  Model  Organizational  Level 


The  experts  identified  broker  reputation,  represented  by  the  below  expert 
comment  as  a  “ sour  view  of  DISA ’s  ability,  ”  as  a  likely  obstacle  during  service 
utilization  and  that  the  presence  of  conflict  would  impact  user  perception  of  whether  the 
“right  decision”  is  made  when  selecting  the  broker.  The  reputation  that  an  individual  or 
group  has  toward  how  a  broker  has  implemented  technology  in  the  past  would  impact 
successful  cloud  computing  implementation  (Bailey  and  Pearson  1983,  Sarker  and 
Valacich,  2010).  DISA  could  reduce  the  impact  of  this  obstacle  through  effective 
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marketing  and  propose  effective  organizational  training.  This  would  result  in  a 


downstream  effect  on  training  culture  and  the  supported  environment. 


DISA  ’.S'  reputation  is  likely  a  road  block  to  service  utilization.  Many  IT/COMM 

personnel ’s  DISA  experience . have  a  sour  view  of  DISA ’s  ability  to  provide  quality 

service  (HBSS),  with  low  latency  (DEPS),  and  ease  of  access  (we’ve  had  CAC 
authentication  blunders).  (Expert  5  Response) 

IT  staff  maturity  was  defined  by  the  experts  as  a  necessary  factor  to  affect 
successful  cloud  computing.  The  experts  agreed  through  model  concurrence  that  this 
would  be  a  hurdle  to  overcome  if  not  developed  through  training.  The  theory  seems  to 
support  that  as  individuals  and  groups  show  an  ‘intention  to  use’  (i.e.  more  users/more 
need)  then  an  increasingly  competent  IT  staff  is  required  to  maintain  the  system 
(Venkatesh  et  al,  2007).  Finally  IT  staff  maturity  would  drive  training  and  the  supported 
environment. 


The  factors  for  successful  cloud  computing  implementations  are  a  ‘good’  customer,  an 
environment  capable  of  supporting  the  implementation,  minimal  environmental  inertia 
(i.e.  minimal  political  churn  preventing  implementation),  and  mature  technical  IQ  of  all 
staff  involved.  (Expert  6  Response) 

Leadership  vision  is  paramount  to  successful  cloud  implementation  and  was 
identified  by  the  expert  response  below  as  a  challenge  that  lacks  innovation.  Industry 
leadership  vision  is  driven  by  business  models  that  are  designed  to  achieve  increased 
market  shares,  where  DoD  leadership  vision  drives  success  based  on  lives  saved  and 
mission  success.  Executing  decisions  and  providing  guidance  based  on  re-vectored  DoD 
visions  may  overcome  this  challenge  (Chidambaram  and  Tung,  2005).  Leadership  vision 
will  affect  system  configurations  and  the  organizational  environment  as  it  becomes  more 
mature,  potentially  until  successful  cloud  computing  implementation  is  realized. 
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Leadership.  Most  military  leaders  have  not  been  promoted  for  being  innovative.  Just  as 
with  our  federal  bureaucracy,  change  comes  very  slowly.  Commercial  adoption  of 
technology  is  driven  by  market  share  and  business  leaders  recognize  that  if  they  don ’t 
move,  they  may  not  be  here  tomorrow.  (Expert  4  Response) 

Training  culture  was  not  established  within  initial  expert  responses,  but  after  pass 
#2  was  incorporated  into  the  models,  and  was  agreed  to  be  a  factor.  Training  culture  was 
present  in  parts  of  all  theories  due  to  its  necessity  and  effect  on  other  variables  such  as 
understanding  (Bailey  and  Pearson,  1983).  Training  culture  would  also  impact  output 
quality  existing  in  the  anchor  constructs  of  TAM3  (Venkatesh  and  Bala,  2008).  This 
process  was  argued  necessary  due  to  the  developmental  nature  of  cloud  in  the  DoD.  It 
would  ensure  that  not  only  users,  but  also  developers  and  maintainers  are  properly 
prepared. 

There  will  need  to  be  a  development  cloud .  There  will  need  to  be  a  learning  process 

on  how  to  write  the  cloud  computing  resources . Cloud  resource  providers  will  need 

to  ensure  that  resources  are  sufficient  to  provide  the  desired  computational  resources  to 
satisfy  the  users.  It  will  not  take  many  instances  of  someone  unable  to  get  the  machines 
that  they  need  before  they  will  give  up  and  simply  buy  their  own  machines.  ( Expert  2 
Response) 

Policy  guides  IT  development  in  both  industry  and  DoD  and  was  defined  by 
experts  as  a  mechanism  that  puts  the  ‘right  stuff  in  the  cloud.  For  successful  cloud 
computing  implementation  to  take  place  strong  governance  and  policy  needs  to  be  in 
place.  Policy  and  guidance,  as  an  external  control  or  ‘facilitating  conditions’  is  a  driver  of 
design  characteristics  which  can  change  a  user’s  perceived  ease  of  use  (Venkatesh  and 
Bala  2008).  The  key  to  policy  is  to  not  impede  the  user  or  organization  so  much  that  it 
drives  them  away  from  using  the  technology.  In  the  DoD  pushing  effective  policy  will  be 
a  significant  challenge  and  will  impose  an  effect  on  the  environment. 
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Policy.  One  of  the  most  significant  challenges  will  be  to  develop  the  information  sharing 
mindset  and  implement  powerful  guidance/standards  to  force  the  issue.  Despite 
numerous  senior  leader  decrees,  directives  and  orders,  organizations  still  think  they 
“own  ”  the  data.  (Expert  4  Response) 


Flexible  configuration  not  only  affects  the  environment,  but  also  impacts 

technology  quality.  Experts  identified  flexible  configuration  as  needed  to  adjust  for 

changing  architectures,  analytic  tools,  change  in  business  model,  and  adjust  for 

proprietary  data  handling  to  add  benefit  and  access  to  all  data/information.  This 

flexibility  is  typically  experienced  in  an  IT  system  as  facilitating  condition  that  increases 

its  effectiveness  by  adjusting  to  user  wants  and  needs  (Venkatesh  et  al,  2003).  If  DISA  is 

flexible  enough  in  their  implementation  of  cloud  then  it  should  be  a  quality  product.  If 

they  do  not  adjust  based  on  user  and  organizational  churn  then  it  could  become  a 

cumbersome  challenge  with  dramatic  effects  on  the  environment. 

Managers  must  understand  that  there  is  no  ‘out  of  the  box  ’  solution  to  reducing  costs, 

increasing  flexibility  or  achieving  sccde  by  buying  something  off  of  the  shelf. . 

Successfully  implementing  cloud  computing  solutions  requires  a  change  in  business 
mindset  that  may  be  far  more  difficult  to  achieve  than  simply  convincing  someone  to  buy 
or  acquire  a  particular  technology.  (Expert  1  Response) 

The  environment  was  affected  directly  and  indirectly  by  the  preceding  6  processes 
and  imposed  an  effect  on  technology  quality  and  productivity  in  other  levels  of  the 
model.  Experts  defined  the  environment  as  the  holistic  implementation  of  all  6  of  those 
processes  discussed  previously  in  conjunction  with  DoD  specific  culture.  The 
environment  construct  is  reflected  in  most  acceptance  theories  impacting  behavioral 
intentions,  use  behavior  (UTAUT),  perceived  usefulness  and  perceived  ease  of  use 
(TAM2)  (Venkatesh  et  al,  2003;  Venkatesh  and  Davis,  200).  Since  the  environment  is  so 
encompassing  in  this  model,  and  affects  users  and  organizations  in  so  many  ways,  it 
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should  be  addressed  as  one  of  the  most  important  processes  to  achieve  successful  cloud 
computing  implementation. 

Contractors  must  have  the  proper  incentives  in  place  to  provide  applications  within  the 
cloud  computing  environment.  Technical:  Environment  should  be  vertically  integrated  to 
provide  seamless  access  at  both  an  Infrastructure  as  a  Service  level  and  a  Platform  as  a 

Sendee  level . Resource  management  must  be  built  into  the  core  of  the 

environment  to  allow  for  the  seamless,  automated  allocation  of  resources  based  on  policy 
and  not  manual  allocation  based  on  trouble  ticket  like  requests  to  a  help  desk.  ( Expert  3 
Response) 

The  final  process  of  economic  factors  in  the  organizational  level  of  the  model  was 

defined  by  experts  as  the  quantifiable  savings  and  the  monetary  value  of  cloud 

computing.  There  was  no  downstream  affect  from  this  process,  but  it  is  affected  by 

productivity  in  the  user  level.  Theory  has  found  that  IT  business  value  is  in  its  cost 

reduction  driven  from  productivity  enhancement  (Melville  et  al,  2004).  In  the  DoD  this  is 

realized  in  conceptualizing  saving  lives  and  maintaining  mission  assurance,  but  hard 

dollar  savings  can  be  captured  through  O  &  M,  hardware,  and  software  savings  driven  by 

higher  rates  of  usage  if  cloud  computing  implementation  is  successful. 

Cloud  computing  enables  you  to  conduct  certain  business  activities  better  and  multiplies  the 
impact  of  better  business  transactions  in  three  distinct  areas;  1)  Productivity  Improvements,  2) 
Lowered  Total  Cost  of  Ownership  (Cost  Avoidance)  and  3)  Hard  Dollar  Savings  through  reduced 
Capital,  Plant,  and  Equipment.  Cloud  computing  is  a  “Combat  Multiplier”  and  that  multiplier  is 
Value.  ( Expert  7  Response) 

Technology  Level 


At  this  level  of  the  model,  DISA  could  pursue  two  paths  for  cloud  computing 
implementation.  The  first  would  be  to  follow  the  process  model  design  and  incorporate 
DoD  development  based  on  the  DoD  culture,  requirements  and  use  cases  if  it  is 
“demanded”.  The  second  development  option  would  be  to  follow  successful  commercial 
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models.  The  below  clarification  of  the  technology  level  processes  should  aid  in  that 
decision. 

Only  two  constructs  were  found  to  be  relevant  within  the  technology  level  of  the 
process  model  (Figure  15).  Defined  requirements  ties  to  technology  quality  and  both  are 
affected  by  the  organizational  level  while  impacting  productivity  at  the  user  level.  The 
effects  in  the  technology  level  of  the  process  model  were  fewer  than  previous  levels.  The 
technology  level  is  critical  due  to  the  impact  it  could  have  on  successful  cloud 
commuting  implementation.  The  ability  to  define  requirements  in  the  DoD  is  a  tricky 
endeavor  that  should  be  addressed  by  leadership. 


Figure  15.  Process  Model  Technology  Level 


The  relationship  between  requirements  definition  and  the  technology  quality 
could  be  a  crossroads  in  the  implementation  path.  Not  accurately  and  clearly  defining 
requirements  could  lead  to  disaster  upon  implementation.  The  experts  identified  that 
requirements  needed  to  be  successful  and  properly  vetted  through  appropriate  channels. 
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Industry  tends  to  creep  along  their  development  path  following  requirements  along  the 
way  and  testing  intermittently  to  ensure  they  are  being  met.  In  the  DoD,  processing 
requirements  can  be  so  cumbersome  that  they  have  changed  or  are  no  longer  effective  by 
the  time  they  are  approved.  Sproles,  2000  postulates  deploying  MOEs  as  an  effective 
tool  to  determine  requirements  impact.  In  the  DoD  requirements  are  a  driver  of  effective 
systems,  leading  to  quality  systems  (technology).  From  this  we  can  speculate  that  MOEs 
in  the  DoD  could  lead  to  quality  technology,  lending  evidence  to  the  linkage  in  the 
model. 

Information  technologies  are  driven  by  the  dynamic  market  trends  and  can  rarely,  if  ever, 
be  predicted  years  in  advance  through  a  centralized  requirements  process.  ”  2)  Outdated 
acquisition  mindsets:  It ’s  actually  the  mindsets  that  aren  7  working.  If  you  study  DoD 
acquisition,  there  are  plenty  of  “processes  ’’for  acquiring  programs  rapidly,  managing 
risk,  and  coping  with  unforeseen  changes,  but  we  rarely  use  them  appropriately  for 
implementing  modern  IT.  (Expert  3  Response) 

Technology  quality  is  the  next  process  in  the  model.  This  process  was  defined  by 
experts  as  innovative  applications,  lending  themselves  to  maintaining  data  integrity, 
security,  reliability  and  near  zero  latency  within  interoperable  systems.  Defined 
requirements  are  external  variables  that  result  from  the  characteristics  of  tasks  and  the 
system  that  influence  users  and  groups  (Zain  et  al,  2005).  Adjustable  requirements  add 
needed  flexibility  via  user  input.  The  resultant  user/client  buy-in  could  lead  to  increased 
productivity  at  the  user  level  of  the  model. 

Innovative  applications  that  users  find  valuable  are  critically  important.  Useless  or 
difficult  to  use  software  will  hinder  cloud  computing  utilization.  ( Expert  3  Response ) 
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User  Level 


The  user  level  consists  of  the  processes  access,  support,  understanding, 
productivity  improvement,  and  support  (Figure  16).  Four  of  these  constructs  exist  in  the 
variance  model  and  are  based  on  the  same  ideas,  theories  and  comments  that  led  to  the 
variables  used.  The  process  productivity  improvement  exists  at  this  level  based  on 
support  from  the  M-TAG  construct  task-technology  fit  of  the  complexity  of  the 
technology.  If  the  users  and  organization  do  not  view  the  complexity  of  the  system  as  an 
obstacle,  then  they  could  show  an  increased  valence  towards  that  technology  (Sarker  and 
Valacich,  2010).  This  could  lead  to  increased  productivity  elevating  the  chances  of 
successful  cloud  computing  implementation. 


Figure  16.  Process  Model  User  Level 
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Conclusion 


If  all  applicable  processes  in  the  model  are  addressed  and  linkages  between  each 
process  followed,  successful  cloud  implementation  can  be  realized.  There  is  a  risk  of 
failure  to  successful  implementation  if  only  a  subset  of  the  model  is  executed.  There  was 
concurrence  on  the  processes  and  links  from  the  expert  panel,  further  cementing  their 
importance.  Further  clarification  of  variables  and  processes  within  these  models  can  be 
achieved  by  analyzing  the  experts'  comments  (Appendix  C). 

This  chapter  showed  the  analysis,  theory  and  models  resulting  from  three  passes 
of  the  Delphi  study  (Appendix  D).  The  first  pass  was  used  to  identify  factors  that,  if 
addressed,  can  achieve  successful  cloud  computing.  Those  variables  were  analyzed  and 
deployed  via  two  models  in  a  second  pass  where  the  experts  clarified  the  findings.  This 
chapter  also  identified  the  variables  that  should  enable  a  successful  cloud  implementation 
strategy  and  answered  research  question  1.  Those  variables  determined  in  this  research, 
led  to  answering  research  question  2,  determining  how  DISA  could  execute 
implementation  and  is  addressed  in  Chapter  V.  Using  the  results  from  the  first  two 
research  questions,  viable  models  were  developed,  while  answering  research  question  3 
from  Chapter  I.  Those  models  were  then  introduced  in  this  chapter.  Also  in  this  chapter, 
supported  theories  of  literature  were  presented.  This  chapter  showed  how  the  models 
were  derived  from  those  theories  and  the  constructs  and  variables  within  those  models 
were  defined.  Concurrence  on  these  models  was  achieved  from  the  experts  via  a  third 
pass. 
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V.  Conclusions  and  Recommendations 


Overview 

The  goal  of  this  research  was  to  develop  models  for  execution  of  a  cloud  solution 
by  DISA.  That  model  development  was  achieved.  This  chapter  now  presents  answers  to 
the  research  questions  and  conclusions  from  this  research.  It  will  also  describe  any 
limitations  encountered  during  the  Delphi  study  or  any  other  areas.  Finally,  it  will 
identify  benefits  of  the  study  and  recommend  areas  for  future  research  to  further  expand 
the  subject  area. 

Limitations 

There  were  some  limitations  to  this  research.  The  sample  size  for  this  research 
was  10  experts,  with  one  dropping  out  for  non-response.  As  with  any  research,  this 
sample  size  could  be  increased,  but  care  should  be  taken  when  reaching  theoretical 
saturation  that  was  mentioned  in  Chapter  III.  An  increased  amount  of  time  between 
contacts  with  panel  members  could  have  enabled  the  researcher  to  go  back  to  the  experts 
for  more  clarification  and  details.  This  additional  data  could  have  provided  additional 
validity  of  the  models.  Finally,  a  survey  was  originally  looked  at  for  accomplishing  this 
research.  This  tool  was  abandoned  due  to  the  cumbersome  process  for  acquiring  a  survey 
control  number  that  would  have  enabled  the  researcher  to  accomplish  the  survey. 
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Research  Question  Answers 


RESEARCH  QUESTION  1: 

What  organizational  variables  or  processes  influence  successful  cloud  computing 
implementation  in  the  DoD? 

Variables  and  processes  that  could  influence  successful  cloud  computing 
implementation  in  the  DoD  are  listed  in  Table  4. 

RESEARCH  QUESTION  2: 

How  could  DISA  implement  a  cloud  solution  ? 

DISA’s  cloud  implementation  process  will  be  critical.  This  research  has  given 
them  two  potential  tools  to  use  prior  to  execution  to  ensure  that  process  is  as  efficient  and 
effective  as  it  allows  them  to  be.  This  research  has  provided,  but  one  piece  of  that 
implementation  process,  the  rest  should  be  dictated  by  how  DISA  answers  their  COAs. 
The  strategy  should  be  flexible  enough  to  adjust  to  technology  and  have  DISA-imposed 
milestones  in  place  in  advance. 

First,  based  on  expert  comments,  “there  will  need  to  be  a  development  cloud.” 
This  development  cloud  should  reflect  true  implementation  by  considering  this  research 
when  building  and  testing  it.  This  could  reduce  any  learning  curve  involved  and  would 
ensure  the  right  structure  is  implemented  that  provides  the  user  and  organizations  with 
the  resources  needed  to  overcome  variables  and  processes  within  the  research  models. 
The  below  expert  comment  reinforces  the  need  for  a  development  cloud  to  include  testing 
the  effects  on  users  and  organizations.  However,  final  implementation  should  reflect  two 
different  paths  to  increase  successfulness. 

It  will  not  take  many  instances  of  someone  unable  to  get  the  machines  that  they  need 
before  they  will  give  up  and  simply  buy  their  own  machines.  (Expert  2  Response) 
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There  are  three  paths  to  successful  cloud  computing  implementation  that  DISA 
could  pursue.  The  first  would  be  to  follow  the  process  model  design  and  incorporate  DoD 
specific  development  based  on  the  DoD  culture,  requirements  and  use  cases  if  it  is 
“demanded”.  This  has  shown  to  be  difficult  and  cumbersome  through  such  technologies 
as  Sharepoint.  The  second  development  option  would  be  to  follow  successful  commercial 
models.  The  last,  and  recommended,  would  be  an  implementation  that  merges  both 
options.  By  taking  into  consideration  successful  commercial  implementation  and 
inserting  DoD  specific  requirements  where  necessary  this  option  seems  to  be  the  most 
attractive.  To  note;  DISA  needs  to  ensure  that  the  ‘olde  think’  of  the  DoD  culture  does 
not  creep  back  into  the  process,  losing  sight  of  the  end  results  that  mirror  industry 
success. 

RESEARCH  QUESTION  3: 

Can  a  "model"  be  developed  that  will  assist  DISA's  strategy  for  successfully 
fielding  cloud  computing  in  the  DoD? 

Two  separate  models  were  developed  and  presented  in  Figures  1 1  and  12, 
respectively,  using  variables  and  processes  gleaned  from  research  question  1. 

Research  Conclusions 

This  research  concentrated  on  user  and  organizational  characteristics.  It  was  done 
with  various  user  level  theories  and  culminated  in  a  group  level  theory  while  showing 
consistency  to  the  variables  throughout.  This  research  concludes  that  those  constructs 
were  consistent  based  on  past  theory  and  expert  comments  acquired  during  this  research. 
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Budget  was  not  prevalent  throughout  literature,  but  was  mentioned  enough  by 
experts  to  give  merit  to  its  importance.  Based  of  expert  responses  it  seems  there  is  a 
disparity  between  how  DoD  budgets  for  new  technology  compared  to  industry.  The 
below  expert  comment  expresses  how  much  industry  spends  on  cloud.  These  budgets  put 
that  disparity  into  perspective. 

DARPA  ’.S' .  2014  budget  submission  was  ~$2.8B  and  depending  on  how  you  slice  and 

dice  them  program  lines,  only  about  a  third  of  that  is  IT  related.  Now,  just  look  at 
Microsoft’s  research  and  development  budget  for  the  period  ending  in  June  2012. 
Microsoft  spent  almost  $10B!  That’s  just  one  company.  Google  spends  a  little  less  than 
$2B  annually  and  Intel  spends  around  $2.5B.  We  simply  cannot  compete  with  those 
numbers  and  expect  to  influence  the  direction  of  the  IT  market  through  direct  investment. 
We  really  need  to  figure  out  how  to  get  on  this  train  rather  than  complain  about  not 
building  the  tracks.  ( Expert  5  Response) 


For  this  reason,  it  seems  conclusive  that  the  budgetary  obstacle  may  be  more 
pertinent  to  DoD  IT  acceptance  and  will  be  advised  of  future  research.  A  further 
conclusion  can  be  reached  that  DoD  implementation  challenges  are  different  from 
industry,  based  on  the  absence  of  these  that  factors  in  theory. 

DoD  could  be  less  likely  to  take  individual  variables  into  account.  DoD  culture 
has  an  innate  tendency  to  force  technologies  into  user  and  organizational  environments 
without  first  addressing  the  obstacles  that  reside  within  those  environments.  If  this 
conclusion  were  to  hold  true,  it  is  less  likely  that  DISA  and  the  DoD  would  adopt  the 
process  model  over  the  variance  model.  An  alternative  to  this  conclusion  would  be  to 
deploy  the  models  at  different  levels.  The  variance  model  could  be  deployed  at  the 
organizational/individual  level  and  the  process  could  be  implemented  at  the  overarching 
strategic  level  i.e.  "Big"  Air  Force  or  DoD. 
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Recommendations  for  Practice 


Understanding  this  research  and  its  ramifications,  if  the  models  are  executed, 
inevitably  leads  an  organization  to  investigate  the  "best  practices"  for  utilizing  this 
research  with  a  cloud  solution.  First,  keep  abreast  of  changes  in  the  cloud  technology 
landscape.  Changes  in  technology  could  force  developers  to  rethink  implementation 
strategies  and  in  turn  change  how  these  models  are  fielded.  Next  managers  and 
developers  should  stay  aware  of  the  programs  and  services  that  their  "cloud"  provides. 
Not  only  does  this  awareness  assist  in  reducing  security  breaches,  but  in  relation  to  this 
research,  it  allows  needed  updates  and  application  changes  to  be  executed  while  realizing 
user  and  organizational  impacts.  Also,  ensure  policy  and  guidance  is  kept  current.  This  is 
pivotal  in  the  DoD  environment  to  ensure  organizations  are  knowledgeable  of  what  the 
"cloud"  has  to  offer  to  the  mission.  This  will  give  them  a  stake  in  the  technology  and 
should  increase  their  desire  to  ensure  success.  Finally,  do  not  rely  on  the  technology 
alone  to  enable  success.  Incorporating  user  and  organizational  variables  to  ensure  the 
technology  is  usable  and  efficient  should  lead  to  achieving  mission  goals. 

Cloud  computing,  like  many  other  technologies,  will  see  success  through  being 
implemented  aggregately  by  encompassing  multiple  programs,  business  functions,  and 
time.  Successful  cloud  solutions  do  not  win  on  their  own,  looking  at  "what  is  in  it"  is  key. 
To  enforce  this  type  of  aggregation  DISA  should  look  at  the  cloud  models  and  services 
they  provide.  DISA  should  implement  multiple  cloud  models  (Private,  Public,  and 
Hybrid)  as  well  as  multiple  services  (IaaS,  PaaS,  SaaS).  This  should  ensure  aggregation 
of  programs  based  on  requirements  driven  by  user  and  organizational  influence  as  well  as 
leadership  vision  and  DoD  directives. 
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Practices  for  managing  cloud  provisioning  and  infrastructure  will  drive  decision 
making,  maintaining  long  term  sustainability,  and  promote  resource  reuse.  Below  are 
questions  to  ask  that  could  drive  "best  practices"  in  cloud  implementation  that  were  posed 
by  the  Software  Sustainability  Institute  in  "Best  Practice  for  Using  Cloud  in  Research" 
(Hong  et  al,  2012).  These  questions  could  also  be  investigated  by  DISA  to  drive  cloud 
practices. 


•  Will  the  cloud  you  choose  be  there  for  as  long  as  DoD  needs  it? 

•  For  commercial  clouds,  what  happens  if  you  don ’t  pay  your  bill?  Is  your  content 
deleted?  Are  you  warned  first? 

•  Does  the  cloud  provider  manage  backups  and,  if  so,  how  often?  If  not,  then  is 
there  a  way  for  you  to  easily  do  backups? 

•  Is  the  help  and  support  offered  by  the  cloud  providers  adequate  for  you? 

•  Is  there  an  SLA  defining  resource  availability,  downtime,  networking  bandwidth, 
etc.  ? 

•  Do  you  have  a  contingency  plan  for  if  your  cloud  were  to  become  unavailable?  Is 
there  another  infrastructure  you  could  use?  Would  you  have  the  time,  money  and 
effort  to  migrate  your  content  ?  What  are  the  consequences  if  there  is  no 
alternative  available? 

•  Are  you  allowed  to  put  your  data  in  the  cloud? 

•  Are  there  any  community  procedures,  institutional  policies  or  legal  frameworks 
you  have  to  comply  with  in  both  hosting  data  on  the  cloud  and  transferring 
applications  and  data  to  and  from  it? 

•  Is  the  use  of  a  public  or  community  cloud  acceptable  to  your  stakeholders? 

•  Does  the  licensing  of  your  application  allow  you  to  deploy  and  use  it  in  the 
cloud? 

•  Do  you  understand  the  licensing  of  your  application  or  will  you  need  to  consult 
with  advisory  bodies  e.g.  OSS-Watch  or  JISCLegal? 

•  Is  the  cloud  you  use  free  to  use  or  will  you  have  to  pay  for  it? 

•  Can  you  estimate  how  much  it  will  cost  you?  Is  this  within  your  financial 
abilities?  Is  this  acceptable  to  your  funders? 

•  How  will  you  pay  for  usage?  Are  you  happy  to  use  your  own  credit  card?  Does 
your  institution  have  a  credit  card  you  can  use  and  would  be  happy  for  you  to  do 
so? 

•  How,  and  how  often,  will  you  monitor  your  resource  usage  to  ensure  you  don ’t 
incur  excessive  charges? 

•  How  will  you  manage  problems  in  porting  or  using  software  on  the  cloud? 

•  How  much  time  will  you  spend  trying  to  get  one  piece  of  software  working  with 
another? 
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•  Do  you  have  a  contingency  plan  in  place  with  alternative  options  to  explore? 

•  When  will  you  decide  that  you ’ve  spent  too  long  and  either  quit  or  explore 
alternative  options  ? 

Recommendations  for  Future  Research 

This  research  is  a  strong  beginning  to  determining  how  to  implement  a  cloud 
computing  solution  with  the  optimal  results.  There  is  additional  research  that  could 
provide  even  further  benefits.  First,  research  could  be  performed  to  determine  any 
correlation  between  how  many  times  a  variable  was  mentioned  by  the  experts  to  how  it 
fit  into  the  models.  This  would  provide  a  broader  understanding  of  the  impact  of  the 
comments  on  how  the  models  were  built.  The  models  have  been  validated  by  industry 
and  DoD  experts,  but  they  could  be  tested  for  statistical  fit  to  the  desired  effect  of 
successful  cloud  computing  implementation.  Further  quantitative  validation  when  the 
solution  is  fielded  would  be  beneficial  for  both  further  phases  of  cloud  implementation  as 
well  as  utilization  when  other  forms  of  technology  are  pursued. 

Further  analysis  addressing  the  multi-level  relationship  of  the  constructs  that 
exists  within  the  two  research  models  would  be  beneficial  to  further  this  research.  A 
quantitative  analysis  of  that  relationship  would  aid  DISA  in  determining  which  factors  to 
put  more  emphasis  on  when  deploying  the  models. 

It  was  noted  in  Chapter  IV  that  technology  theories  did  not  conclusively  provide 
supporting  evidence  regarding  budget,  but  was  mentioned  as  a  critical  construct  by  the 
experts.  Research  into  this  disparity  could  be  further  pursued  to  determine  its  true 
validity.  It  could  be  that  the  experts  identified  that  construct  knowing  that  cloud 
computing  implementation  was  only  germane  to  the  DoD  in  this  research. 
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Conclusion 


This  chapter  provided  limitations  to  this  research,  conclusions,  and  further 
research.  The  capstone  to  this  research  lies  in  the  implementation  of  the  models 
exclusively  by  DISA  before  the  solution  is  deployed.  In  today's  environment  of  tighter 
budgets,  reduced  manpower,  and  cumbersome  acquisitions,  an  efficient  cloud  computing 
solution  is  critical.  This  research  may  provide  DoD  IT  managers,  practitioners,  and 
developers  a  valuable  tool  to  ensure  this  efficiency  is  achieved  in  a  timely  and  acceptable 
timeline.  In  the  end,  the  mission  will  not  fail,  but  properly  fielded  technology  will 
increase  mission  effectiveness  exponentially. 
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Appendix  A:  Construct  Tie-In 
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Appendix  B:  Factor  Comment  Support 


Factor 

Comment 

Understanding 

Incentive  to  use  cloud  computing 

Common  Understanding 

User  understanding 

Recognize  the  business  aspects. .  .productivity  improvements, 
lowered  total  cost  of  ownership 
and  hard  dollar  savings 

Understanding  and  definition 

Manager  Understanding  of  metrics 

Weak  support  model 

Not  everything  belongs  in  the  cloud 

Discovery. .  .what  is  present  and  where  to  go 

Outdated  Acquisition  Mindset 

Cyber  Threat  "Awareness"  vs  "Understanding" 

Inability  to  quantify  gains 

Old  think 

Unbiased  brokering  service 

Unclear/ambiguous  requirements 

Leaders  not  understanding  the  benfits  short  of  a  ROI 

Customer  knowing  what  they  want. .  .reliability,  security, 
efficiencies,  effectiveness 

Leverage  Cloud  Benefits 

Learning  process  to  write  RFPs,  proposals,  and  contracts 

Change  the  incentive  structure 

Implement  aggressiv  timelines 

Simplify  scope  and  requirements 

If  DoD  wants  to  be  an  enterprise  then  think  and  behave  as  one 

Security 

Trust 

Security 

Security 

Security  priority 

DoD  requires  security 

Risk 

Different  security  protoocls  in  DoD 

Budget 

Sufficient  Time  for  Investment 

Cost 

Min.  overhead 

Government  Budget  Cycle 

DoD  funding 

Illusion  of  a  vast  monetary  ROI 
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Pices  driven  up  due  to  inlfated  units  because  of  geographic 
location 

Leverage  a  larger  IT  budget 

Long  term  contracts  with  exit  clauses  for  non-performance 

Access 

Aggregate  Programs  into  the  cloud 

Limit  Restrictions 

Access 

Data  access 

Sharing  Resources 

Min.  barrier  to  entry 

Application  restrictions 

Ability  to  implement  across  NIPR  and  SIPR 

DoD  requires  compartmentalization 

Computational  services 

Ensure  sufficient  resources 

Managed  service  approach  vs.  buying  individual  capability 

Reliability 

Define  Intangibles  (Scalability  and  Llexibility) 

Interconnectivity 

Reliability 

Performance 

Interconnectivity 

Levels  of  performance  expectations 

DoD  requires  stability 

Environmnent 

Cloud  as  a  Business  Process 

"Good"  customer  environment 

Environment  supports  implementation,  min.  environmental 
inertia,  and  technical  IQ  of  staff 

Excessive  beaurocracy  creating  excessive  timeline 

Innovative  applications 

Adequate  advertising/marketing 

DISA  reputation 

Lack  of  "good"  customer  base  for  implementation  and 
utilization 

Organizational  chum 

Mature  business  model/enterprise  business  model 

Service  advocacy 

Unsupportive  customer 

Technically  immature  environment 

Lack  of  organizational  support 

DoD  is  change  adverse 

Indusctry  used  best  effort 

Incentives  for  industry  companies  to  build  applications 

Sense  of  urgency 

Leadership 
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Maintain  situational  awareness  to  the  extent  that  DoD  has  grown 

DoD  acquires  solution  before  defining  the  problem 

Industry  defines  problem  from  vsarious  views  that  provide 
highest  value 

Industry  uses  long  term  partners  with  required  espertise 

DoD  has  doubt. .  .they  must  "check  and  control  everything" 

Seek  out  the  little  guys 

DoD  needs  to  get  out  of  cloud  business  and  offer  cloud  as 
contract  service 

DoD  partners  and  agencies  should  partner  together  to  deliver 
cloud  cap.  for  max  reuse 

Standards 

No  standard  between  DoD  agencies 

No  standard  SLAs  that  provide  "best  value" 

Policy 

Conflict  of  interest,  DISA  is  the  cloud  broker  and  cloud  provider 

Outdated  Policies 

Acquisition  process 

Migration/evolution  from  existing  PoRs 

Evolution  Roadmap 

Provided  service  level  vs.  service  level  paid  for 

Complicated  ATO  and  DAA  porcess  to  be  on  networks 

DoD  use  cases  and  implementation  are  different 

C  &  A,  IA  controls,  and  ATO'd  environments  delay 
implementation 

Undustry  focuses  on  SLAs  not  what  lies  "behind  the  door" 

Development  cloud  and  an  ATO  cloud 

Agencies  honor  community  C  &  A  accreditation  approval 

Commercial  based  SLAs 

Standard  levels  of  service 
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Appendix  C:  Expert  Comments 


Below  are  the  responses  from  pass  one  listed  question-by-question.  All  responses  have 
been  stripped  of  data  indentifying  the  respondent.  Please  read  them  carefully  taking  these  inputs 
into  consideration  when  filling  out  Part  B. 

EXPERT  QUESTION  1  : 

What  factors  should  be  considered  for  successful  cloud  computing  implementation  ? 

Expert  1  Response 

Cloud  Computing  is  a  Business  Process  more  than  a  technology:  Fundamentally, 
implementing  cloud  computing  solutions  is  about  business  processes,  not  buying  technologies. 
Managers  must  understand  that  there  is  no  ‘out  of  the  box’  solution  to  reducing  costs,  increasing 
flexibility  or  achieving  scale  by  buying  something  off  of  the  shelf.  It  is  true  that  this  business 
model  is  only  possible  because  of  great  technological  improvements,  particularly  in  the  areas  of: 
persistent  communications,  increased  bandwidth,  high-performance  commodity  processing 
hardware  and  massive  distributed  storage.  Ultimately,  technology  is  only  a  necessary,  but  not 
sufficient  condition  for  cloud  computing  success.  Successfully  implementing  cloud  computing 
solutions  requires  a  change  in  business  mindset  that  may  be  far  more  difficult  to  achieve  than 
simply  convincing  someone  to  buy  or  acquire  a  particular  technology. 

Programs  win  in  the  aggregate,  not  in  isolation:  Another  critical  piece  of  the  puzzle  is 
recognizing  that  success  with  cloud  generally  comes  from  the  aggregation  of  implementing  cloud 
practices  over  many  programs  and  business  functions.  Especially,  if  the  metric  for  success  is  cost 
savings,  then  this  factor  holds  true  even  more.  Transitioning  any  particular  program  to  a  cloud 
solution  will  not  likely  generate  any  cost  savings  if  the  cloud  solution  only  hosts  that  one 
program.  For  example,  the  most  basic  reason  that  IAAS  works  is  because  of  multitenancy.  If  a 
company  implements  an  IAAS  solution,  but  only  hosts  one  program  on  that  architecture,  then  that 
individual  program  carries  all  of  the  capital  expenditure  burden  for  not  only  it’s  application 
components,  but  also  for  the  infrastructure.  Without  multitenancy  in  IAAS  solutions,  the  system 
goes  underutilized  and  the  unit  cost  per  computation  goes  up.  If  that  architecture  hosts  multiple 
program  simultaneously  and  performs  adequate  load  sharing  (another  semi  fundamental  attribute 
of  cloud  architectures)  then  those  capital  expenses  can  be  spread  across  multiple  business  lines 
and  programs  reducing  the  total  burden. 

“The  aggregate”  is  not  just  about  multiple  programs,  but  also  about  time:  Another 
seemingly  fundamental  aspect  of  cloud  computing  is  that  there  will  likely  be  upfront 
transition/implementation  costs  that  will  require  management  to  treat  IT  as  an  investment  rather 
than  simply  as  quick  fix.  This  is  especially  difficult  in  the  government  where  promises  of  future 
savings  are  rarely  fulfilled.  To  implement  a  successful  cloud  computing  architecture,  managers 
must  realize  that  there  will  be  costs  to  build/test/standardize/manage  infrastructure,  recode/retest 
software,  and  many  other  upfront  costs  that  will  need  to  be  addressed.  Cost  savings  will  likely 
only  manifest  themselves  after  a  sufficient  period  of  time  to  recuperate  the  initial  investment.  The 
decision  gets  complicated  because  for  the  initial  period  of  time,  the  cost  of  continuing  with  the 
established  architecture  will  likely  be  cheaper  than  paying  for  the  costs  associated  with 
transitioning  to  the  cloud.  That  cost  delta  takes  time  to  overcome. 
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Defining  the  intangible  metrics  for  success:  Cost  is  the  easiest  and  most  quantifiable 
metric,  but  flexibility  and  scalability  are  likely  as,  or  perhaps  more,  important.  Unfortunately,  it  is 
very  difficult  to  articulate  the  “requirement”  for  either  of  these  areas.  It  is  easy  to  spot  when 
previous  decisions  impose  severe  restrictions  to  current  operations,  but  it’s  much  more  difficult  to 
determine  which  decisions  today  will  create  the  biggest  ripples  down  the  road.  Hindsight  is  20/20 
and  the  expedient  needs  often  outweigh  the  long-term  best  answers.  Fortunately,  we  have  many 
commercial  companies,  such  as  Amazon  and  Google,  (and  also  simply  the  Internet  at  large)  as 
thriving  examples  of  what  properly  implemented  flexibility  and  scalability  can  achieve. 

Trust:  This  is  the  most  important  factor  of  all.  Cloud  computing  is  about  sharing, 
otherwise  there  would  be  no  need  to  move  beyond  the  stove -piped  programs  that  we  are  so  used 
to.  At  least  in  the  government,  our  entire  acquisition  system  is  based  on  hierarchy  and 
accountability.  The  program  manager  is  responsible  for  the  execution  of  their  entire  system.  In 
the  old  model,  trust  is  established  through  control  (as  if  the  two  were  truly  synonymous).  Yet,  the 
modern  IT  ecosystem  does  not  resemble  this  model.  For  example,  Netflix  hosts  video  content  on 
Amazon.com.  Amazon  presents  those  feeds  to  Tier  1  ISPs  who  send  them  onto  many  other  ISPs. 
Your  ISP  serves  them  through  your  Motorola  Surfboard  to  your  Cisco  Router  to  your  Dell 
Laptop,  running  Microsoft  Windows  for  playback  in  a  Google  Chrome  browser.  Netflix  only 
‘owns’  a  very  small  aspect  of  that  system  and  must  trust  (to  a  reasonable  degree)  all  of  the  other 
non-affiliated  components  along  the  way.  Much  of  the  trust  model  in  this  example  is  established 
through  Service  Level  Agreements  (SLAs)  and  mangers  must  grow  to  accept  that  third-party 
providers  will  perform  to  the  degree  outlined  in  those  SLAs. 

Expert  2  Response 

The  framework  and  implementation  needs  to  be  absent  of  restrictions  on  what  can  run. 
The  user  has  a  problem  that  they  want  to  solve,  and  the  cloud  computing  infrastructure  has  to 
make  it  as  easy  (or  nearly  as  easy)  to  implement  as  on  dedicated  hardware.  We  settled  on  Amazon 
EC2  because  we  could  run  basically  whatever  we  wanted/needed  to,  which  is  not  the  case  in 
Microsoft's  or  Google's  offering. 

The  users  have  to  confidence  that  they  will  have  access  to  the  resources  when  they  need 
them.  When  you  buy  a  rack  of  servers  for  your  project,  they  are  'yours'  and  you  can  (in  theory) 
use  them  at  any  time  instantly.  Reality  is  different  but  the  sense  of  control  is  there.  I  have  started 
a  large  number  of  Amazon  EC2  machines  (>100  simultaneous  8 -core  machines)  and  never  had  a 
problem,  so  I  think 
that  whatever  I  need  I  can  get. 

Expert  3  Response 

Incentives:  Organizations  must  have  the  proper  incentives  to  utilize  cloud  computing. 
Contractors  must  have  the  proper  incentives  in  place  to  provide  applications  within  the  cloud 
computing  environment.  Technical:  Environment  should  be  vertically  integrated  to  provide 
seamless  access  at  both  a  Infrastructure  as  a  Service  level  and  a  Platform  as  a  Service  level.  Bad 
choices  of  mismatched  products  at  each  level  of  the  stack  will  lead  to  poor  adoption.  Resource 
management  must  be  built  into  the  core  of  the  environment  to  allow  for  the  seamless,  automated 
allocation  of  resources  based  on  policy  and  not  manual  allocation  based  on  trouble  ticket  like 
requests  to  a  help  desk. 

Expert  4  Response 
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Common  understanding.  There  are  many  terms  being  used  to  describe  what  "cloud" 
is/isn't  which  has  caused  a  significant  amount  of  confusion.  A  significant  portion  of  the  populace 
only  think  of  the  “cloud”  from  Apple,  Google,  Amazon  or  IBM  commercial  offerings  to  store  and 
access  data.  They  don’t  understand  that  there  are  utility  and  analytic  clouds  as  well.  The  picture 
gets  muddier  when  other  terms  like  "net-centric,"  "SOA,"  "IaaS,"  "PaaS,"  "SaaS,"  etc.  are 
discussed. 


Interconnectivity.  Just  as  cellular  service,  email  and  Internet  providers  evolved  through  a 
phase  where  it  was  difficult  to  talk  across  the  proprietary  networks  without  significant  costs, 
cloud  architectures  must  do  the  same. 

Data  access.  There  isn't  enough  money  to  convert  existing  data  stores  to  Accumulo/Big 
Table  so  we  need  to  be  able  to  access  and  integrate  data  from  "legacy"  systems  M2M  with  near 
zero  data  latency.  Our  warfighters  have  come  to  expect  the  same  kind  of  unhindered  information 
access  in  their  professional  work  that  they  enjoy  in  their  private  lives. 

Expert  5  Response 

Reliability:  Solution  needs  to  be  up  all  the  time 

Performance:  Low  latency  from  the  standpoint  of  uploads,  downloads,  and  use  of  the 
cloud  for  computing/applications 

Security:  Traditional  IA  security  as  well  as  data  integrity 

Cost:  Cost  needs  to  be  reasonable.  If  we  can’t  save  money  with  the  cloud  solution,  then 
why  use  it? 

User  understanding  -  What  do  the  user’s  know  about  cloud  computing?  Does  it  meet  or 
beat  their  expectations?  Do  they  understand  non-monetary  benefits  (e.g.,  data  access  anywhere  in 
the  world,  etc) 

Expert  6  Response 

The  factors  for  successful  cloud  computing  implementations  are  a  ‘good’  customer,  an 
environment  capable  of  supporting  the  implementation,  minimal  environmental  inertia  (i.e. 
minimal  political  churn  preventing  implementation),  and  mature  technical  IQ  of  all  staff  involved. 

Expert  7  Response 

Using  cloud  computing  is  more  of  a  business  decision  than  a  technology  decision.  Cloud 
computing  enables  you  to  conduct  certain  business  activities  better  and  multiplies  the  impact  of 
better  business  transactions  in  three  distinct  areas;  1)  Productivity  Improvements,  2)Lowered 
Total  Cost  of  Ownership  (Cost  Avoidance)  and  3)Hard  Dollar  Savings  through  reduced  Capital, 
Plant,  and  Equipment.  Cloud  computing  is  a  “Combat  Multiplier”  and  that  multiplier  is  Value.  To 
implement  successfully  you  must  have  a  strong  Policy  and  Governance  Structure,  A  Business 
Model/Process  Architecture,  and  a  Solution  Architecture. 

Expert  8  Response 

Understanding  and  definition-  across  the  board  the  term  "cloud"  has  different 
definitions  and  even  when  the  definitions  "appear"  similar,  the  intent  and  what  is  included  can 
be  substantially  different.  Commercial  vs  Army  vs  USAF  vs  DoD-  no  standard.  Means 
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SLAs(service  level  agreements)  and  pricing  are  all  over  the  map  vs  clear  standards  that  can  be 
evaluated  and  priced  to  ensure  "best  value"  and  common  levels  of  predictable  delivery. 

EXPERT  QUESTIONS  2: 

What  are  key  concents  for  successful  cloud  computing  utilization  ? 

Expert  1  Response 

Unfortunately,  many  of  the  “factors”  that  1  identified  above  are  also  the  primary 
concerns.  Take,  for  instance,  an  example  where  two  or  more  users  agree  to  share  the  cost  of  a 
common  hardware  infrastructure.  Who  gets  priority  when  there  is  contention  for  resources?  This 
may  be  something  that  users  can  agree  to  in  the  design  phase,  but  it  may  be  much  more  difficult 
for  the  “losing”  program  to  explain  to  their  customers  or  their  supervisors  when  their  program 
does  not  perform  in  operations  as  it  would  have  if  the  underlying  infrastructure  had  been 
dedicated  and  purposefully  built  for  their  sole  use. 

Properly  understanding  the  statistical  nature  of  all  aspects  of  cloud  computing  will  help 
managers  make  sound  long-term,  cost/performance  trades,  but  these  sound  decisions  in  the  past 
may  not  pass  the  test  when  the  rubber  meets  the  road.  I  often  ask  people  a  couple  of  simple 
questions  to  highlight  this  phenomenon.  First,  I  ask  when  was  the  last  time  their  Gmail  account 
was  down?  Most  will  say  never.  Second,  I’ll  ask  when  was  the  last  time  their  work  e-mail  was 
down,  most  will  remember  these  outages  quite  vividly.  Yet  the  times  when  Gmail  or  Amazon 
Web  Services  do  go  down  it  is  a  huge  news  event.  For  some  reason  we  seem  inherently 
comfortable  with  mistakes,  so  long  as  they  are  our  own,  but  we  fail  to  capitalize  on  the 
opportunities  for  better  performance  when  someone  else  may  impose  mistakes  upon  us,  even  if 
there  will  be  far  fewer  mistakes  overall.  This  mindset  needs  to  go  if  we  want  to  be  successful  in 
this  arena. 

Expert  2  Response 

Technically,  the  overhead  needs  to  be  kept  to  a  minimum  and  the  barrier  to  entry  needs 
to  be  kept  low  enough  that  any  application  could  feasibly  be  'moved  to  the  cloud'.  For  example,  1 
was  presented  with  computational  resources  for  a  particular  project  where  the  physical  machines 
were  sufficient.  However,  they  required  me  to  use  a  platform,  namely  Oracle  Grid  Engine,  to 
manage  the  software  that  was  not  a  good  fit  for  my  application.  The  result  was  a  loss  of  time  and 
money. 


Programmatically,  for  commercial  cloud  computing,  I  know  that  I  can  get  the  machines 
that  I  need  and  know  how  much  to  pay  for  them  (X  cents  per  hour,  I  know  how  many  hours  I  am 
using,  and  at  the  end  of  the  month,  Amazon  sends  me  a  bill).  For  a  DoD  cloud  computing 
implementation,  my  concern  is  that  it  will  be  wrapped  with  so  many  gatekeepers  and  layers  of 
bureaucracy  that  it  will  be  almost  impossible  to  get  the  machines  I  need  in  a  timely  manner,  when 
1  need  them,  and  at  a  project  cost  that  is  reasonable. 

Expert  3  Response 

Innovative  applications  that  users  find  valuable  are  critically  important.  Useless  or 
difficult  to  use  software  will  hinder  cloud  computing  utilization. 

Expert  4  Response 


78 


Clear  Objectives.  "Cloud"  technologies  offer  tremendous  promise;  however,  it  seems  that 
the  "buzz"  has  overtaken  critical  evaluation  of  what  is  to  be  achieved.  What  is  the  "cloud"  to 
provide?  Is  it  for  data  center  consolidation,  large  data  analytics,  web-services,  cyber-security, 
etc.? 


Policy.  One  of  the  most  significant  challenges  will  be  to  develop  the  information  sharing 
mindset  and  implement  powerful  guidance/standards  to  force  the  issue.  Despite  numerous  senior 
leader  decrees,  directives  and  orders,  organizations  still  think  they  “own”  the  data. 

Security.  With  improved  access  and  availability  of  data/information,  the  importance  of 
maintaining  security  and  integrity  of  the  data  becomes  paramount. 

Interconnectivity.  A  lot  of  companies  are  having  a  hard  time  adjusting  to  a  new  business 
model  for  DoD  information  systems.  Many  are  proposing  to  simply  build  more  proprietary  data 
handling  architectures  and  analytic  tools  with  no  real  added  benefit  of  providing  ubiquitous 
access  to  all  data/information. 

Expert  5  Response 

Adequate  advertising/marketing  to  ensure  the  customer  understands  the  utility  of  the 
cloud  and  the  DoD  brokering  process. 

For  the  DoD,  a  big  concern  is  likely  the  fact  that  DISA  is  the  broker  for  cloud  as  directed 
by  the  DoD  CIO,  and  at  the  same  time,  DISA  is  a  cloud  provider.  This  is  an  inherent  conflict  of 
interest  that  will  raise  concern  when  a  DISA  cloud  service  is  chosen  when  the  user  doesn’t  feel 
it’s  the  right  decision. 

DISA’s  reputation  is  likely  a  road  block  to  service  utilization.  Many  IT/COMM 
personnel’s  DISA  experience,  whether  with  Flost  Base  Security  System  (HBSS),  DoD  Enterprise 
Email  (DEE,  US  Army  the  first  adopters),  or  Defense  Enterprise  Portal  Services  (DEPS  - 
SharePoint  services),  have  a  sour  view  of  DISA’s  ability  to  provide  quality  service  (HBSS),  with 
low  latency  (DEPS),  and  ease  of  access  (we’ve  had  CAC  authentication  blunders)  is  an  area  of 
concern. 

Expert  6  Response 

My  main  concerns  are  lack  of  a  ‘good’  customer  base  both  for  implementation  and 
utilization,  organizational  churn,  and  a  weak  support  model. 

Expert  7  Response 

Security-  You  must  pick  the  right  computing  model  for  security  to  be  as  effective  as 
possible.  Do  not  select  a  public  cloud  and  expect  it  to  be  as  secure  as  a  trusted  private  cloud. 

Not  everything  belongs  in  a  cloud.  The  most  effective  implementations  are  enterprise¬ 
wide  common  business  tasks  with  high  demand  and  high  transaction  rates. 

Most  implementation  fail  because  they  did  not  have  a  mature  business  model/enterprise 
business  model  from  the  start. 

Expert  8  Response 

Technical  ability  to  execute  is  not  the  major  challenge-  two  major  issues  evolve 
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around  (1)  "discovery"-  what  is  present  and  what  is  to  move  to  a/the  cloud  (2)  levels  of 
performance-  expectations-  goes  back  to  defining  and  establishing  what  is  to  be  delivered 
for  the  agreed  upon  level  of  effort/price. 

EXPERT  QUESTION  3: 

What  main  obstacles  would  hinder  successful  cloud  computing  implementation? 

Expert  1  Response 

The  government  budget  cycle:  It  works  for  acquisition  programs  that  span  many  years, 
but  it  not  well  suited  to  ill  or  undefinable  requirements  that  are  subject  to  change  rapidly.  To  steal 
a  quote  from  my  M1LC0M  paper  “DoD  acquisition  managers  are  under  constant  pressure  to 
maintain  currency  across  their  enterprises  and  meet  ever  changing  requirements  over  an 
increasingly  complex  infrastructure.  It  is  extremely  difficult  to  achieve  such  lofty  goals  under 
layers  of  bureaucracy  and  a  six-year  Planning,  Programming,  Budgeting,  and  Execution  (PPBE) 
cycle.  Information  technologies  are  driven  by  the  dynamic  market  trends  and  can  rarely,  if  ever, 
be  predicted  years  in  advance  through  a  centralized  requirements  process.”  2)  Outdated 
acquisition  mindsets:  It’s  actually  the  mindsets  that  aren’t  working.  If  you  study  DOD 
acquisition,  there  are  plenty  of  “processes”  for  acquiring  programs  rapidly,  managing  risk,  and 
coping  with  unforeseen  changes,  but  we  rarely  use  them  appropriately  for  implementing  modern 
IT. 


Outdated  policies:  This  probably  flows  from  the  previous  obstacle,  since  the  mindsets 
create  the  mental  schema’s  that  inform  the  policies,  but  it’s  real  nonetheless.  The  Federal  CIO 
office  has  made  quite  a  few  important  strides  in  the  area  of  cloud  and  now  the  rest  of  the 
government  needs  to  catch  up...  especially  the  DoD.  The  D1ACAP  policies  practically 
discourage  change  and  force  vendors  to  modify  their  commercial  implementations  to  such 
extremes  that  we  become  the  sole  customer  for  many  products.  As  soon  as  our  path  forks  from 
the  broader  market,  we  are  doomed  to  foot  the  bill  for  all  of  the  associated  costs...  we  lose  all  of 
the  economies  of  scale.  We  are  our  often  our  own  worst  enemy  here. 

The  threat  of  “cyber”:  We  have  seen  a  tremendous  increase  in  the  “awareness”  of  the 
cyber  threat  over  the  last  five  years,  but  awareness  and  understanding  are  two  entirely  different 
things.  Fear  plus  technological  illiteracy  equal  irrationality.  Rather  than  embracing  new 
opportunities,  I’m  afraid  that  fear  of  the  unknown  will  cause  us  to  further  hunker  down  in  the  old, 
comfortable  ways  of  doing  business  (ones  that  are  often  technologically  much  less  secure,  but 
organizationally  more  accountable). 

Expert  2  Response 

The  inability  to  run  necessary  applications  (in  their  current  form)  such  as  restrictions  on 
language/resources  available. 

An  inability  to  quantify  the  gains  (in  time  and/or  money)  in  implementing  a  solution 
using  the  cloud. 

Expert  3  Response 

I  believe  that  the  acquisition  process  of  not  only  the  cloud  computing  environment  itself 
but  for  the  applications  themselves  is  the  main  obstacle  to  successful  cloud  computing 
implementation.  The  acquisition  of  the  cloud  computing  environment  must  ensure  a  true  open 
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environment  for  applications  and  not  an  environment  that  locks  in  certain  industry  players  or 
teams  and  locks  out  small  players.  New  application  are  more  likely  to  come  from  new  or  hungry 
companies  as  opposed  to  the  large  contractors  -  which  will  attempt  to  game  the  acquisition 
process.  Attempts  to  port  legacy  software  may  hinder  the  success  of  cloud  computing  because  it 
will  be  expensive.  The  current  incentive  structure  for  the  incumbents  with  the  existing  contracts 
are  for  it  to  be  difficult  and  expensive. 

Expert  4  Response 

Old  think.  Too  many  policies  from  acquisition,  IT  management,  IC  vs.  DoD,  large 
Programs  of  Record,  etc.  aren’t  enabled  to  rapidly  develop,  transition  and  evolve  to  maximize  the 
advantages  of  “cloud”  technologies. 

Service  advocacy.  Senior  leaders  appear  reluctant  to  commit  to  this  “new”  technology 
without  some  kind  of  demonstration  but  without  their  “buy-in”  it  is  difficult  to  solicit  for  data 
access,  acquire  funds  or  start  breaking  glass  on  how  we  currently  collect,  manage,  evaluate  and 
distribute  information. 

Migration/evolution  from  existing  PoRs.  There  are  a  number  of  extremely  large 
programs  that  could  benefit  significantly  in  cost,  performance  and  schedule  if  they  adopted 
“cloud”  technologies.  The  issue  is  that  they  are  large  PoRs  with  very  sizable  contractual 
commitments  and  are  either  unwilling  or  unable  to  embrace  the  opportunities.  An  example  of  this 
is  the  JMS  program.  Current  cost  of  the  system  is  approaching  $750M  with  no  end  in  sight. 
Implementing  a  “cloud-based”  solution  could  be  done  for  approximately  $125M  in  24  months 
with  senior  leader  advocacy. 

Evolution  Roadmap.  The  DNI  has  developed  a  strategy.  The  DoD  CIO  has  a  strategy  as 
does  each  IC  agency.  The  challenge  is  that  implementation  is  currently  being  downward  directed 
and  the  path  to  interconnected  systems  is  so  dynamic  that  developers  are  having  a  difficult  time 
staying  in  synch  with  the  enterprise. 

Expert  5  Response 

DoD  funding,  DoD  cost  for  SIPR  solutions  (note  that  there  are,  I  believe,  two  commercial 
SIPR  cloud  providers,  but  will  be  in  direct  competition  with  DISA). 

Unbiased  cloud  service  brokering  processes. 

Requirements  gathering  and  vetting  and  ultimately  translated  into  the  right  service,  right 
vendor,  right  cost,  etc. 

The  ability  to  implement  private,  public,  and  hybrid  cloud  solutions  across  2  security 
enclaves  (NIPR  &  SIPR)  and  the  need  for  a  JWICs  solution. 

Expert  6  Response 

The  main  obstacles  are  unclear  /  ambiguous  requirements,  an  unsupportive  customer,  a 
technically  immature  environment  (both  environment  and  IT  IQ),  and  lack  of  overall 
organizational  support. 
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Expert  7  Response 

The  expectation  of  a  vast  monetary  ROI  from  cloud  computing  is  an  illusion.  The  proper 
measure  of  a  successful  implementation  is  VALUE  and  how  value  should  be  measured  becomes 
the  key  critical  driver.  Cloud  computing  is  delivering  a  service  and  to  be  successful  the  service 
must  be  of  value  to  the  customer.  The  truth  of  the  matter  is  cloud  services  could  cost  more  to 
deliver  than  the  legacy  system  it  is  replacing.  However,  if  the  service  provided  is  valuable  to  the 
customers  using  it,  the  environment  will  deliver  value  through  higher  rates  of  usage,  productivity 
improvements  and  other  “soft”  benefits.  Classic  ROI  does  not  do  well  in  a  cloud  environment. 
Some  leaders  will  not  understand  that. 

Expert  8  Response 

Customer/client  actually  knowing  what  they  want/what  they  expect  for  the  price/level 
of  effort  and  to  establish  long  term  partnerships  to  allow  for  continued 
growth/effectiveness/efficiencies.  Many  times  the  client/customer  is  not  sure  of  what  they 
currently  have  and  therefore  not  sure  of  the  business  case  for  change-  how  do  they  ensure  they 
get  what  they  need  and  do  so  in  a  more  cost  efficient,  reliable  manner-  with  the  level  of 
security  and  future  capability  that  is  required. 

EXPERT  QUESTION  4: 

What  are  the  differences  between  the  way  industry  implements  a  cloud  solution  to  the 

way  DoD  should? 

Expert  1  Response 

The  DOD  is  not  risk  averse,  its  change  averse...  otherwise  we  wouldn’t  accept  all  of  the 
risks  associated  with  our  current  IT  architectures.  Industry  is  far  better  at  managing  risk,  when  it 
comes  to  IT  services  than  the  DOD.  We  often  eschew  private  enterprise  because  of  their  desire 
for  profit  and  we  assume  that  they  “cut  comers”  on  security.  I  have  found  just  the  opposite  to  be 
true  for  many  established  cloud  computing  vendors.  In  the  cloud  computing  space,  there  are  not 
significant  barriers  preventing  consumers  from  switching  from  one  provider  to  another.  A 
sufficient  security  breech  could  result  in  a  tremendous  loss  of  revenue  as  consumers  make  that 
change.  These  financial  realities  make  security  a  top  priority. 

Expert  2  Response 

Industry  tends  to  run  on  a  best  effort.  The  DoD  would  require  stability,  security,  and 
compartmentalization  of  the  cloud  in  order  to  protect  assets  and  information. 

Many  DOD  projects  do  not  just  have  an  external  security  concern,  but  also  an  internal 
one.  Since  I  do  development,  what  I  mn  on  my  servers  is  something  that  I  should  not  have  to  go 
through  a  huge  effort  to  get  onto  the  cloud  servers.  When  I  have  access  to  a  rack  of  servers, 
correctly  separated  from  the  rest  of  the  network,  then  I  can  put  software  on  it  during  development, 
but  there  would  seem  to  be  issues  in  doing  that  with  a  shared  resource  in  a  DoD  environment. 
Also,  it  is  not  the  case  that  development  can  be  done  on  a  noncloud  computing  environment, 
approved  (getting  an  ATO)  and  then  putting  it  on  the  cloud. 

The  cost  allocation  for  computational  resources  is  also  very  different.  A  commercial 
project  or  a  DoD  contract  project  sets  aside  a  certain  amount  of  ODCs  and  they  are  used  to  buy 
equipment.  Using  a  cloud  computing  solution  confuses  the  situation  since  they  are  closer  to 
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services,  and  so  there  needs  to  be  a  shift  in  the  way  that  the  money  is  spent,  and  project  managers, 
COTRs,  contracting  officers  need  to  understand  it. 

Expert  3  Response 

Industry  has  much  larger  numbers  of  companies  to  build  cloud  applications  and  these 
companies  have  incentives  to  do  so.  DoD  has  a  relatively  tiny  number  and  these  companies  have 
no  incentive  other  than  the  labor  dollars  they  will  get  paid  to  do  so.  Development  labor  is  a  small 
fraction  of  the  revenue  of  these  companies  -  so  an  environment  that  reduces  their  O&M  revenue 
will  make  cloud  computing  very  difficult  for  these  companies  to  truly  get  on  board  with. 

Expert  4  Response 

Sense  of  urgency.  Time  is  money  in  the  business  world.  If  a  solution  can  save  money 
while  simultaneously  saving  time,  they’re  much  more  willing  to  take  the  risks.  Additionally, 
businesses  are  constantly  challenged  by  their  competitors  so  they  don’t  have  the  luxury  of 
exhaustively  studying  their  options  like  the  DoD. 

Risk.  Industry  tends  to  build  a  little,  test  a  little  and  adjust  the  course  of  development. 
DoD,  on  the  other  hand,  spends  so  much  time  investigating  the  options  that  by  the  time  a 
requirements  document  is  presented  for  bids,  the  requirements  will  frequently  be  unable  to  reflect 
the  current  state  of  technology.  There  is  some  merit  to  this  because  DoD  generally  measures  risk 
by  lives  saved  versus  dollars  spent  but  it’s  almost  to  the  point  of  stagnating  progress. 

Leadership.  Most  military  leaders  have  not  been  promoted  for  being  innovative.  Just  as 
with  our  federal  bureaucracy,  change  comes  very  slowly.  Commercial  adoption  of  technology  is 
driven  by  market  share  and  business  leaders  recognize  that  if  they  don’t  move,  they  may  not  be 
here  tomorrow. 

Expert  5  Response 

From  the  Operations  Community  is  maintaining  situational  awareness  to  the  granularity 
the  DoD  has  grown  to  expect. 

Actual  service  level  provided  vice  service  level  paid  for.  For  example,  DISA  hosts 
applications  in  the  Defense  Enterprise  Computing  Centers  (DECCs),  and  these  systems  have  a 
Mission  Assurance  Category  (MAC)  applied  to  them  which  has  an  associated  cost.  If  the  Marine 
Corp  Online  systems  goes  down,  a  general  officer  to  general  officer  call  is  made,  and  it  is  treated 
as  a  MAC  I  (mission  critical,  restoration  time  within  a  few  hours,  etc)  even  though  the  USMC 
isn’t  paying  for  that  level  of  service  (which  has  no  MAC  associated  with  it,  which  means  we  have 
say  4  days  to  address  the  problem). 

DoD  has  different  security  rules/processes  than  commercial  providers,  and  I  think  ours 
tend  to  be  more  strict,  which  may  drive  up  our  cost,  increase  implementation  time,  etc. 

Traditionally,  the  DoD  isn’t  about  making  money  and  some  of  our  organizations  are 
inflated  due  to  geographic  location.  This  drives  the  price  up,  and  I  think  it  will  be  hard  for  us  to 
compete  financially  with  commercial  providers. 

DoD  often  has  complicated  ATO  and  DAA  processes  to  operate  systems  on  a  DoD 
network.  The  commercial  sector  may  not  have  this  issue.  It  certainly  affects  our  ability  to  be  agile 
around  customer  needs. 
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Expert  6  Response 

The  DoD  has  been  adopting  capabilities  in  a  manner  that  Industry  has  been  delivering 
them.  The  DoD  use  cases  and  implementation  are  different  than  the  Industry  use  cases.  In 
addition,  the  emphasis  on  C&A,  1A  controls,  and  ATO’d  environments  delay  most  if  not  timely 
implementations. 

Expert  7  Response 

DoD  seems  to  acquire  a  solution  before  they  have  defined  the  problem  it’s  supposed  to 
solve.  Industry  will  define  the  problem  from  various  perspectives  and  then  explore  solutions  that 
deliver  the  highest  value  for  the  most  reasonable  price. 

Expert  8  Response 

Commercial  industry  finds  a  long  term  partner  who  is  a  proven  expert  in  the  field  (has 
the  experience  and  the  capability),  hands  over  the  mission,  looks  for  long  term  associations 
where  benefits  are  shared  (e.g.  the  commercial  provider  saves  the  client  money  and  makes 
money  from  that  savings-  direct  benefits  from  quality  work  and  providing  efficiencies  and 
improved  performance).  Commercial  clients  focus  on  the  SLAs  and  not  what  is  behind  the 
door-  the  reason  they  hired  an  expert  to  do  the  work.  DoD  has  a  since  of  doubt-  they  must 
check  and  control  everything  vs.  a  managed  services  approach  based  on  delivery  of  a  product 
of  the  quality  and  capability  required  and  at  a  price  incentivized  by  performance  and 
efficiency/cost  savings. 

EXPERT  QUESTION:  5 

How  could  those  obstacles  and  differences  be  overcome? 

Expert  1  Response 

Embrace  sequestration!  Our  fiscally  constrained  environment  will  force  us  to  chart  new 
territory  (at  least  new  to  the  DOD).  The  good  news  is  that  we  have  many  examples  of  how  private 
industry  is  successfully  leveraging  cloud  implementations  to  their  great  benefit.  All  we  need  to  do 
is  follow.  That’s  hard  for  us,  especially  since  we  are  the  industry  driver  in  most  other  aspects  of 
the  DOD  (fighter  airplanes,  most  satellites,  tanks,  etc).  This  isn’t  so  in  the  information  technology 
space.  We  are  a  drop  in  the  overall  bucket.  For  example,  according  to  DARPA’s  website 
(www.darpa.mil/WorkArea/DownloadAsset.aspx7id-2147486441),  their  2014  budget  submission 
was  ~$2.8B  and  depending  on  how  you  slice  and  dice  their  program  lines,  only  about  a  third  of 
that  is  IT  related.  Now,  look  at  just  Microsoft’s  research  and  development  budget  for  the  period 
ending  in  June  2012.  Microsoft  spent  almost  $  1  OB !  That’s  just  one  company.  Google  spends  a 
little  less  than  $2B  annually  and  Intel  spends  around  $2.5B.  We  simply  cannot  compete  with 
those  numbers  and  expect  to  influence  the  direction  of  the  IT  market  through  direct  investment. 
We  really  need  to  figure  out  how  to  get  on  this  train  rather  than  complain  about  not  building  the 
tracks. 

Expert  2  Response 

There  will  need  to  be  a  development  cloud  (with  development  network  or  unconnected 
connectivity)  and  an  Approved  To  Operate  cloud.  There  will  need  to  be  a  learning  process  on 
how  to  write  the  cloud  computing  resources  into  RFPs,  proposals,  and  contracts.  Cloud  resource 
providers  will  need  to  ensure  that  resources  are  sufficient  to  provide  the  desired  computational 
resources  (and  memory  and  disk  space)  to  satisfy  the  users.  It  will  not  take  many  instances  of 
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someone  unable  to  get  the  machines  that  they  need  before  they  will  give  up  and  simply  buy  their 
own  machines. 

Expert  3  Response 

I'm  not  sure  how  to  overcome  these  obstacles.  The  incentive  structure  probably  has  to 
somehow  be  changed. 

Expert  4  Response 

Implement  aggressive  timelines  with  manageable  and  measurable  milestones  to  deliver 
real  capabilities  incrementally.  Don’t  try  to  build  the  whole  house  at  once. 

Seek  out  the  little  guys.  There  are  hundreds  of  very  small  businesses  that  are  hungry  to 
deliver  capabilities  very  cheaply  and  very  quickly.  I  don’t  think  anyone  would  equate  the  large 
DoD  corporations  as  agile  or  nimble  nor  are  they  willing  to  sign  a  contract  for  six  months  for 
$200k.  That’s  not  how  they  became  multi-billion  dollar  companies.  That’s  how  you  build  an  F- 
35. 


Simplify  scope  and  requirements.  Again,  build  a  little  and  test  a  little  to  build  momentum. 
Expert  5  Response 

The  DoD  needs  to  get  out  of  the  Cloud  business,  and  offer  it  as  a  contract  service.  Don’t 
try  to  compete  by  providing  a  private  cloud  on  NIPR  and  all  classes  of  clouds  on  SIPR.  DoD 
(DISA)  should  establish  security  and  NetOps  criteria  and  simply  broker  the  services  (have 
contracts  available  for  users,  and  assist  in  matching  user  requirements  to  providers). 

Expert  6  Response 

DoD  partners  and  agencies  should  partner  together  to  build  a  community  that  delivers 
cloud  capabilities  that  can  be  easily  shared  for  maximum  reuse.  The  community  should  also 
engage  in  following  community  practices  for  ATO  for  ‘type  accreditation’  where  any 
organization  and  agency  that  is  developing  capabilities  that  achieves  accreditation  should  be  able 
to  apply  these  capabilities.  Unfortunately,  every  agency  has  its  own  rules  for  C&A  but  should 
honor  a  community  accreditation  approval. 

Expert  7  Response 

In  my  view  DoD  is  moving  in  the  right  direction.  The  obstacles  faced  by  DoD  fall  for  the 
most  part  in  thinking  and  behaving.  The  business  enterprise  of  DoD  is  not  that  different  from  a 
large  multinational/global  company.  Both  must  have  HR,  Logistics,  Finance,  Infrastructure,  and 
other  common  business  functions.  DoD  has  a  culture/people/institutional  barrier  to  think  and 
behaving  as  a  business  enterprise  in  areas  where  that  think  could  benefit  them  the  most.  If  you 
want  to  be  an  enterprise,  think  and  behave  as  one. 

Expert  8  Response 

Managed  services  approach  vs.  buying  individual  capability,  long  term  contracts 
with  clear  exit  clauses  for  non-performance,  commercially  based  SLAs,  and  standard  levels  of 
service  across  the  organizations  vs.  constantly  treating  every  org  is  a  one-off. 
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Appendix  D:  Delphi  Questionnaires 


CLOUD  COMPUTING  IMPLEMENTATION  ORGANIZATIONAL  SUCCESS  IN  THE 

AIR  FORCE 

Demographic  Questionnaire 


Name: 

Date: 

Title: 

Organization: 

Instructions 

This  questionnaire  collects  specific  demographic  information,  but  only  generalities  will  be  used 
when  the  data  is  presented.  All  data  will  be  aggregated  or  otherwise  processed  before  public 
release.  I  understand  that  the  names  and  associated  data  I  collect  must  be  protected  at  all  times, 
only  be  known  to  the  researchers,  and  managed  according  to  the  AFIT  interview  protocol. 

All  data  will  only  be  handled  by  the  researcher  and  advising  committee.  Please  feel  free  to  contact 
me  with  any  questions  at  937-541-8169  or  e-mail  corey.perkins@us.af.mil. 

Please  answer  the  following  questions 

1.  In  your  current  position  and  in  regards  to  cloud  computing  are  you  a  developer, 
practitioner,  or  manager? 

2.  In  your  current  position  are  you  associated  with  Industry,  the  Air  Force  or  the  Army? 

3.  How  many  years  of  experience  do  you  have  in  the  IT  realm? 

4.  How  many  years  of  experience  do  you  have  in  the  research  area  of  cloud  computing 
success?  Delphi  Questions 

Greetings,  welcome,  and  again  thank  you  for  participating!  Below  are  five  questions 
addressing  areas  of  research  in  regards  to  cloud  computing  success.  Once  received  and  an 
alyzed  the  2nd  pass  of  questions  will  follow.  Please  read  the  questions  carefully, 
elaborate  as  much  asneeded  for  each  question  and  be  as  specific  as  possible.  Feel  free 
to  ask  for  explanation  onanything  you  do  not  understand. 

Provide  responses  to  corey.perkins@us.af.mil  NLT  21  June,  2013. 
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QUESTION  1 : 

What  factors  do  you  see  as  considering  successful  cloud  computing  implementation  ? 


QUESTION  2: 

What  are  your  key  concerns  for  successful  cloud  computing  utilization  ? 


QUESTION  3: 

What  are  the  mean  obstacles  you  envision  hindering  successful  cloud  computing 
implementation  ? 


QUESTION  4: 

What  do  you  see  as  differences  between  the  way  industry  implements  a  cloud  solution  to  the 
way  DoD  should? 


QUESTION  5: 

How  would  you  overcome  those  obstacles  and  differences? 


FOLLOW  UP  QUESTION: 

Is  there  anything  else  not  covered  that  could  be  added? 
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CLOUD  COMPUTING  IMPLEMENTATION  ORGANIZATIONAL  SUCCESS  IN  THE 

AIR  FORCE 


Delphi  2nd  Pass 

Greetings,  welcome,  and  again  thank  you  for  participating!  This  the  second  round  of  the  Delphi 
study.  In  this  round,  you  are  given  the  responses  from  round  one,  our  categorization  of  those 
responses,  as  well  as  two  models  that  attempt  to  capture  causality  and  process  flow.  What  we  ask 
in  this  round  is  that  you  review  the  responses,  categorization,  and  models,  and  then  answer  the 
nine  numbered  questions.  Depending  on  the  desired  level  of  effort,  this  round  should  take 
between  one  and  two  hours.  In  the  interest  of  preserving  your  time,  we  made  some  of  the 
questions  optional  (noted  as  “EXTRA  CREDIT”),  as  we  realize  they  may  take  more  than  the 
allotted  time. 

We  appreciate  your  patience  as  we  analyzed  the  data  from  Round  1 .  Upon  completion  of  Round 
2  (which  should  not  take  as  much  time  as  the  intervening  weeks  between  Round  1  and  Round  2), 
the  finalized  model(s)  will  be  resent  out  for  a  3rd  pass — essentially  a  confirmation  that  we  have 
converged  on  a  final  model.  Feel  free  to  ask  for  explanation  on  anything  you  do  not  understand. 

As  before,  all  data  will  be  aggregated  or  otherwise  processed  before  public  release.  Names  and 
associated  data  will  be  protected  at  all  times,  only  known  to  the  researchers,  and  managed 
according  to  the  AFIT  interview  protocol.  All  data  will  only  be  handled  by  the  researcher  and 
advising  committee.  Please  feel  free  to  contact  me  with  any  questions  at  937-541-8169.  Provide 
responses  to  corey.perkins@us.af.mil  NLT  30  August,  2013. 

Part  A  : 


Below  are  the  responses  from  pass  one  listed  question-by-question.  All  responses  have  been 
stripped  of  data  indentifying  the  respondent.  Please  read  them  carefully  taking  these  inputs  into 
consideration  when  filling  out  Part  B . 

EXPERT  QUESTION  1  : 

What  factors  should  be  considered  for  successful  cloud  computing  implementation? 

Expert  1  Response 

Cloud  Computing  is  a  Business  Process  more  than  a  technology:  Fundamentally,  implementing 
cloud  computing  solutions  is  about  business  processes,  not  buying  technologies.  Managers  must 
understand  that  there  is  no  ‘out  of  the  box’  solution  to  reducing  costs,  increasing  flexibility  or 
achieving  scale  by  buying  something  off  of  the  shelf.  It  is  true  that  this  business  model  is  only 
possible  because  of  great  technological  improvements,  particularly  in  the  areas  of:  persistent 
communications,  increased  bandwidth,  high-performance  commodity  processing  hardware  and 
massive  distributed  storage.  Ultimately,  technology  is  only  a  necessary,  but  not  sufficient 
condition  for  cloud  computing  success.  Successfully  implementing  cloud  computing  solutions 
requires  a  change  in  business  mindset  that  may  be  far  more  difficult  to  achieve  than  simply 
convincing  someone  to  buy  or  acquire  a  particular  technology. 

Programs  win  in  the  aggregate,  not  in  isolation:  Another  critical  piece  of  the  puzzle  is 
recognizing  that  success  with  cloud  generally  comes  from  the  aggregation  of  implementing  cloud 
practices  over  many  programs  and  business  functions.  Especially,  if  the  metric  for  success  is  cost 
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savings,  then  this  factor  holds  true  even  more.  Transitioning  any  particular  program  to  a  cloud 
solution  will  not  likely  generate  any  cost  savings  if  the  cloud  solution  only  hosts  that  one 
program.  For  example,  the  most  basic  reason  that  IAAS  works  is  because  of  multitenancy.  If  a 
company  implements  an  IAAS  solution,  but  only  hosts  one  program  on  that  architecture,  then  that 
individual  program  carries  all  of  the  capital  expenditure  burden  for  not  only  it’s  application 
components,  but  also  for  the  infrastructure.  Without  multitenancy  in  IAAS  solutions,  the  system 
goes  underutilized  and  the  unit  cost  per  computation  goes  up.  If  that  architecture  hosts  multiple 
program  simultaneously  and  performs  adequate  load  sharing  (another  semi  fundamental  attribute 
of  cloud  architectures)  then  those  capital  expenses  can  be  spread  across  multiple  business  lines 
and  programs  reducing  the  total  burden. 

“The  aggregate”  is  not  just  about  multiple  programs,  but  also  about  time:  Another 
seemingly  fundamental  aspect  of  cloud  computing  is  that  there  will  likely  be  upfront 
transition/implementation  costs  that  will  require  management  to  treat  IT  as  an  investment  rather 
than  simply  as  quick  fix.  This  is  especially  difficult  in  the  government  where  promises  of  future 
savings  are  rarely  fulfilled.  To  implement  a  successful  cloud  computing  architecture,  managers 
must  realize  that  there  will  be  costs  to  build/test/standardize/manage  infrastructure,  recode/retest 
software,  and  many  other  upfront  costs  that  will  need  to  be  addressed.  Cost  savings  will  likely 
only  manifest  themselves  after  a  sufficient  period  of  time  to  recuperate  the  initial  investment.  The 
decision  gets  complicated  because  for  the  initial  period  of  time,  the  cost  of  continuing  with  the 
established  architecture  will  likely  be  cheaper  than  paying  for  the  costs  associated  with 
transitioning  to  the  cloud.  That  cost  delta  takes  time  to  overcome. 

Defining  the  intangible  metrics  for  success:  Cost  is  the  easiest  and  most  quantifiable 
metric,  but  flexibility  and  scalability  are  likely  as,  or  perhaps  more,  important.  Unfortunately,  it  is 
very  difficult  to  articulate  the  “requirement”  for  either  of  these  areas.  It  is  easy  to  spot  when 
previous  decisions  impose  severe  restrictions  to  current  operations,  but  it’s  much  more  difficult  to 
determine  which  decisions  today  will  create  the  biggest  ripples  down  the  road.  Hindsight  is  20/20 
and  the  expedient  needs  often  outweigh  the  long-term  best  answers.  Fortunately,  we  have  many 
commercial  companies,  such  as  Amazon  and  Google,  (and  also  simply  the  Internet  at  large)  as 
thriving  examples  of  what  properly  implemented  flexibility  and  scalability  can  achieve. 

Trust:  This  is  the  most  important  factor  of  all.  Cloud  computing  is  about  sharing, 
otherwise  there  would  be  no  need  to  move  beyond  the  stove -piped  programs  that  we  are  so  used 
to.  At  least  in  the  government,  our  entire  acquisition  system  is  based  on  hierarchy  and 
accountability.  The  program  manager  is  responsible  for  the  execution  of  their  entire  system.  In 
the  old  model,  trust  is  established  through  control  (as  if  the  two  were  truly  synonymous).  Yet,  the 
modern  IT  ecosystem  does  not  resemble  this  model.  For  example,  Netflix  hosts  video  content  on 
Amazon.com.  Amazon  presents  those  feeds  to  Tier  1  ISPs  who  send  them  onto  many  other  ISPs. 
Your  ISP  serves  them  through  your  Motorola  Surfboard  to  your  Cisco  Router  to  your  Dell 
Laptop,  running  Microsoft  Windows  for  playback  in  a  Google  Chrome  browser.  Netflix  only 
‘owns’  a  very  small  aspect  of  that  system  and  must  trust  (to  a  reasonable  degree)  all  of  the  other 
non-affiliated  components  along  the  way.  Much  of  the  trust  model  in  this  example  is  established 
through  Service  Level  Agreements  (SLAs)  and  mangers  must  grow  to  accept  that  third-party 
providers  will  perform  to  the  degree  outlined  in  those  SLAs. 
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Expert  2  Response 

The  framework  and  implementation  needs  to  be  absent  of  restrictions  on  what  can  run. 
The  user  has  a  problem  that  they  want  to  solve,  and  the  cloud  computing  infrastructure  has  to 
make  it  as  easy  (or  nearly  as  easy)  to  implement  as  on  dedicated  hardware.  We  settled  on  Amazon 
EC2  because  we  could  run  basically  whatever  we  wanted/needed  to,  which  is  not  the  case  in 
Microsoft's  or  Google's  offering. 

The  users  have  to  confidence  that  they  will  have  access  to  the  resources  when  they  need 
them.  When  you  buy  a  rack  of  servers  for  your  project,  they  are  'yours'  and  you  can  (in  theory) 
use  them  at  any  time  instantly.  Reality  is  different  but  the  sense  of  control  is  there.  1  have  started 
a  large  number  of  Amazon  EC2  machines  (>100  simultaneous  8 -core  machines)  and  never  had  a 
problem,  so  I  think  that  whatever  I  need  I  can  get. 

Expert  3  Response 

Incentives:  Organizations  must  have  the  proper  incentives  to  utilize  cloud  computing. 
Contractors  must  have  the  proper  incentives  in  place  to  provide  applications  within  the  cloud 
computing  environment.  Technical:  Environment  should  be  vertically  integrated  to  provide 
seamless  access  at  both  a  Infrastructure  as  a  Service  level  and  a  Platform  as  a  Service  level.  Bad 
choices  of  mismatched  products  at  each  level  of  the  stack  will  lead  to  poor  adoption.  Resource 
management  must  be  built  into  the  core  of  the  environment  to  allow  for  the  seamless,  automated 
allocation  of  resources  based  on  policy  and  not  manual  allocation  based  on  trouble  ticket  like 
requests  to  a  help  desk. 

Expert  4  Response 

Common  understanding.  There  are  many  terms  being  used  to  describe  what  "cloud" 
is/isn't  which  has  caused  a  significant  amount  of  confusion.  A  significant  portion  of  the  populace 
only  think  of  the  “cloud”  from  Apple,  Google,  Amazon  or  IBM  commercial  offerings  to  store  and 
access  data.  They  don’t  understand  that  there  are  utility  and  analytic  clouds  as  well.  The  picture 
gets  muddier  when  other  terms  like  "net-centric,"  "SOA,"  "IaaS,"  "PaaS,"  "SaaS,"  etc.  are 
discussed. 

Interconnectivity.  Just  as  cellular  service,  email  and  Internet  providers  evolved  through  a 
phase  where  it  was  difficult  to  talk  across  the  proprietary  networks  without  significant  costs, 
cloud  architectures  must  do  the  same. 

Data  access.  There  isn't  enough  money  to  convert  existing  data  stores  to  Accumulo/Big 
Table  so  we  need  to  be  able  to  access  and  integrate  data  from  "legacy"  systems  M2M  with  near 
zero  data  latency.  Our  warfighters  have  come  to  expect  the  same  kind  of  unhindered  information 
access  in  their  professional  work  that  they  enjoy  in  their  private  lives. 

Expert  5  Response 

Reliability:  Solution  needs  to  be  up  all  the  time 

Performance:  Low  latency  from  the  standpoint  of  uploads,  downloads,  and  use  of  the 
cloud  for  computing/applications 

Security:  Traditional  IA  security  as  well  as  data  integrity 
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Cost:  Cost  needs  to  be  reasonable.  If  we  can’t  save  money  with  the  cloud  solution,  then 
why  use  it? 

User  understanding  -  What  do  the  user’s  know  about  cloud  computing?  Does  it  meet  or 
beat  their  expectations?  Do  they  understand  non-monetary  benefits  (e.g.,  data  access  anywhere  in 
the  world,  etc) 

Expert  6  Response 

The  factors  for  successful  cloud  computing  implementations  are  a  ‘good’  customer,  an 
environment  capable  of  supporting  the  implementation,  minimal  environmental  inertia  (i.e. 
minimal  political  churn  preventing  implementation),  and  mature  technical  IQ  of  all  staff  involved. 

Expert  7  Response 

Using  cloud  computing  is  more  of  a  business  decision  than  a  technology  decision.  Cloud 
computing  enables  you  to  conduct  certain  business  activities  better  and  multiplies  the  impact  of 
better  business  transactions  in  three  distinct  areas;  1)  Productivity  Improvements,  2)Lowered 
Total  Cost  of  Ownership  (Cost  Avoidance)  and  3)Hard  Dollar  Savings  through  reduced  Capital, 
Plant,  and  Equipment.  Cloud  computing  is  a  “Combat  Multiplier”  and  that  multiplier  is  Value.  To 
implement  successfully  you  must  have  a  strong  Policy  and  Governance  Structure,  A  Business 
Model/Process  Architecture,  and  a  Solution  Architecture. 

Expert  8  Response 

Understanding  and  definition-  across  the  board  the  term  "cloud"  has  different 
definitions  and  even  when  the  definitions  "appear"  similar,  the  intent  and  what  is  included  can 
be  substantially  different.  Commercial  vs  Army  vs  USAF  vs  DoD-  no  standard.  Means  SLAs 
(service  level  agreements)  and  pricing  are  all  over  the  map  vs  clear  standards  that  can  be 
evaluated  and  priced  to  ensure  "best  value"  and  common  levels  of  predictable  delivery. 

EXPERT  QUESTIONS  2: 

What  are  key  concerns  for  successful  cloud  computing  utilization  ? 

Expert  1  Response 

Unfortunately,  many  of  the  “factors”  that  I  identified  above  are  also  the  primary 
concerns.  Take,  for  instance,  an  example  where  two  or  more  users  agree  to  share  the  cost  of  a 
common  hardware  infrastructure.  Who  gets  priority  when  there  is  contention  for  resources?  This 
may  be  something  that  users  can  agree  to  in  the  design  phase,  but  it  may  be  much  more  difficult 
for  the  “losing”  program  to  explain  to  their  customers  or  their  supervisors  when  their  program 
does  not  perform  in  operations  as  it  would  have  if  the  underlying  infrastructure  had  been 
dedicated  and  purposefully  built  for  their  sole  use. 

Properly  understanding  the  statistical  nature  of  all  aspects  of  cloud  computing  will  help 
managers  make  sound  long-term,  cost/performance  trades,  but  these  sound  decisions  in  the  past 
may  not  pass  the  test  when  the  rubber  meets  the  road.  I  often  ask  people  a  couple  of  simple 
questions  to  highlight  this  phenomenon.  First,  I  ask  when  was  the  last  time  their  Gmail  account 
was  down?  Most  will  say  never.  Second,  I’ll  ask  when  was  the  last  time  their  work  e-mail  was 
down,  most  will  remember  these  outages  quite  vividly.  Yet  the  times  when  Gmail  or  Amazon 
Web  Services  do  go  down  it  is  a  huge  news  event.  For  some  reason  we  seem  inherently 
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comfortable  with  mistakes,  so  long  as  they  are  our  own,  but  we  fail  to  capitalize  on  the 
opportunities  for  better  performance  when  someone  else  may  impose  mistakes  upon  us,  even  if 
there  will  be  far  fewer  mistakes  overall.  This  mindset  needs  to  go  if  we  want  to  be  successful  in 
this  arena. 

Expert  2  Response 

Technically,  the  overhead  needs  to  be  kept  to  a  minimum  and  the  barrier  to  entry  needs 
to  be  kept  low  enough  that  any  application  could  feasibly  be  'moved  to  the  cloud'.  For  example,  1 
was  presented  with  computational  resources  for  a  particular  project  where  the  physical  machines 
were  sufficient.  However,  they  required  me  to  use  a  platform,  namely  Oracle  Grid  Engine,  to 
manage  the  software  that  was  not  a  good  fit  for  my  application.  The  result  was  a  loss  of  time  and 
money. 


Programmatically,  for  commercial  cloud  computing,  1  know  that  1  can  get  the  machines 
that  I  need  and  know  how  much  to  pay  for  them  (X  cents  per  hour,  I  know  how  many  hours  I  am 
using,  and  at  the  end  of  the  month,  Amazon  sends  me  a  bill).  For  a  DoD  cloud  computing 
implementation,  my  concern  is  that  it  will  be  wrapped  with  so  many  gatekeepers  and  layers  of 
bureaucracy  that  it  will  be  almost  impossible  to  get  the  machines  I  need  in  a  timely  manner,  when 
1  need  them,  and  at  a  project  cost  that  is  reasonable. 

Expert  3  Response 

Innovative  applications  that  users  find  valuable  are  critically  important.  Useless  or 
difficult  to  use  software  will  hinder  cloud  computing  utilization. 

Expert  4  Response 

Clear  Objectives.  "Cloud"  technologies  offer  tremendous  promise;  however,  it  seems  that 
the  "buzz"  has  overtaken  critical  evaluation  of  what  is  to  be  achieved.  What  is  the  "cloud"  to 
provide?  Is  it  for  data  center  consolidation,  large  data  analytics,  web-services,  cyber-security, 
etc.? 


Policy.  One  of  the  most  significant  challenges  will  be  to  develop  the  information  sharing 
mindset  and  implement  powerful  guidance/standards  to  force  the  issue.  Despite  numerous  senior 
leader  decrees,  directives  and  orders,  organizations  still  think  they  “own”  the  data. 

Security.  With  improved  access  and  availability  of  data/information,  the  importance  of 
maintaining  security  and  integrity  of  the  data  becomes  paramount. 

Interconnectivity.  A  lot  of  companies  are  having  a  hard  time  adjusting  to  a  new  business 
model  for  DoD  information  systems.  Many  are  proposing  to  simply  build  more  proprietary  data 
handling  architectures  and  analytic  tools  with  no  real  added  benefit  of  providing  ubiquitous 
access  to  all  data/information. 

Expert  5  Response 

Adequate  advertising/marketing  to  ensure  the  customer  understands  the  utility  of  the 
cloud  and  the  DoD  brokering  process. 

For  the  DoD,  a  big  concern  is  likely  the  fact  that  DISA  is  the  broker  for  cloud  as  directed 
by  the  DoD  CIO,  and  at  the  same  time,  DISA  is  a  cloud  provider.  This  is  an  inherent  conflict  of 
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interest  that  will  raise  concern  when  a  D1SA  cloud  service  is  chosen  when  the  user  doesn’t  feel 
it’s  the  right  decision. 

DISA’s  reputation  is  likely  a  road  block  to  service  utilization.  Many  IT/COMM 
personnel’s  D1SA  experience,  whether  with  Host  Base  Security  System  (HBSS),  DoD  Enterprise 
Email  (DEE,  US  Army  the  first  adopters),  or  Defense  Enterprise  Portal  Services  (DEPS  - 
SharePoint  services),  have  a  sour  view  of  DISA’s  ability  to  provide  quality  service  (HBSS),  with 
low  latency  (DEPS),  and  ease  of  access  (we’ve  had  CAC  authentication  blunders)  is  an  area  of 
concern. 

Expert  6  Response 

My  main  concerns  are  lack  of  a  ‘good’  customer  base  both  for  implementation  and 
utilization,  organizational  churn,  and  a  weak  support  model. 

Expert  7  Response 

Security-  You  must  pick  the  right  computing  model  for  security  to  be  as  effective  as 
possible.  Do  not  select  a  public  cloud  and  expect  it  to  be  as  secure  as  a  trusted  private  cloud. 

Not  everything  belongs  in  a  cloud.  The  most  effective  implementations  are  enterprise¬ 
wide  common  business  tasks  with  high  demand  and  high  transaction  rates. 

Most  implementation  fail  because  they  did  not  have  a  mature  business  model/enterprise 
business  model  from  the  start. 

Expert  8  Response 

Technical  ability  to  execute  is  not  the  major  challenge-  two  major  issues  evolve  around  (1) 
"discovery"-  what  is  present  and  what  is  to  move  to  a/the  cloud  (2)  levels  of  performance- 
expectations-  goes  back  to  defining  and  establishing  what  is  to  be  delivered  for  the  agreed  upon 
level  of  effort/price. 

EXPERT  QUESTION  3: 

What  main  obstacles  would  hinder  successful  cloud  computing  implementation? 

Expert  1  Response 

The  government  budget  cycle:  It  works  for  acquisition  programs  that  span  many  years, 
but  it  not  well  suited  to  ill  or  undefinable  requirements  that  are  subject  to  change  rapidly.  To  steal 
a  quote  from  my  M1LCOM  paper  “DoD  acquisition  managers  are  under  constant  pressure  to 
maintain  currency  across  their  enterprises  and  meet  ever  changing  requirements  over  an 
increasingly  complex  infrastructure.  It  is  extremely  difficult  to  achieve  such  lofty  goals  under 
layers  of  bureaucracy  and  a  six-year  Planning,  Programming,  Budgeting,  and  Execution  (PPBE) 
cycle.  Information  technologies  are  driven  by  the  dynamic  market  trends  and  can  rarely,  if  ever, 
be  predicted  years  in  advance  through  a  centralized  requirements  process.”  2)  Outdated 
acquisition  mindsets:  It’s  actually  the  mindsets  that  aren’t  working.  If  you  study  DOD 
acquisition,  there  are  plenty  of  “processes”  for  acquiring  programs  rapidly,  managing  risk,  and 
coping  with  unforeseen  changes,  but  we  rarely  use  them  appropriately  for  implementing  modern 
IT. 

Outdated  policies:  This  probably  flows  from  the  previous  obstacle,  since  the  mindsets 
create  the  mental  schema’s  that  inform  the  policies,  but  it’s  real  nonetheless.  The  Federal  CIO 
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office  has  made  quite  a  few  important  strides  in  the  area  of  cloud  and  now  the  rest  of  the 
government  needs  to  catch  up...  especially  the  DoD.  The  D1ACAP  policies  practically 
discourage  change  and  force  vendors  to  modify  their  commercial  implementations  to  such 
extremes  that  we  become  the  sole  customer  for  many  products.  As  soon  as  our  path  forks  from 
the  broader  market,  we  are  doomed  to  foot  the  bill  for  all  of  the  associated  costs...  we  lose  all  of 
the  economies  of  scale.  We  are  our  often  our  own  worst  enemy  here. 

The  threat  of  “cyber”:  We  have  seen  a  tremendous  increase  in  the  “awareness”  of  the 
cyber  threat  over  the  last  five  years,  but  awareness  and  understanding  are  two  entirely  different 
things.  Fear  plus  technological  illiteracy  equal  irrationality.  Rather  than  embracing  new 
opportunities,  I’m  afraid  that  fear  of  the  unknown  will  cause  us  to  further  hunker  down  in  the  old, 
comfortable  ways  of  doing  business  (ones  that  are  often  technologically  much  less  secure,  but 
organizationally  more  accountable). 

Expert  2  Response 

The  inability  to  run  necessary  applications  (in  their  current  form)  such  as  restrictions  on 
language/resources  available. 

An  inability  to  quantify  the  gains  (in  time  and/or  money)  in  implementing  a  solution 
using  the  cloud. 

Expert  3  Response 

I  believe  that  the  acquisition  process  of  not  only  the  cloud  computing  environment  itself 
but  for  the  applications  themselves  is  the  main  obstacle  to  successful  cloud  computing 
implementation.  The  acquisition  of  the  cloud  computing  environment  must  ensure  a  true  open 
environment  for  applications  and  not  an  environment  that  locks  in  certain  industry  players  or 
teams  and  locks  out  small  players.  New  application  are  more  likely  to  come  from  new  or  hungry 
companies  as  opposed  to  the  large  contractors  -  which  will  attempt  to  game  the  acquisition 
process.  Attempts  to  port  legacy  software  may  hinder  the  success  of  cloud  computing  because  it 
will  be  expensive.  The  current  incentive  structure  for  the  incumbents  with  the  existing  contracts 
are  for  it  to  be  difficult  and  expensive. 

Expert  4  Response 

Old  think.  Too  many  policies  from  acquisition,  IT  management,  IC  vs.  DoD,  large 
Programs  of  Record,  etc.  aren’t  enabled  to  rapidly  develop,  transition  and  evolve  to  maximize  the 
advantages  of  “cloud”  technologies. 

Service  advocacy.  Senior  leaders  appear  reluctant  to  commit  to  this  “new”  technology 
without  some  kind  of  demonstration  but  without  their  “buy-in”  it  is  difficult  to  solicit  for  data 
access,  acquire  funds  or  start  breaking  glass  on  how  we  currently  collect,  manage,  evaluate  and 
distribute  information. 

Migration/evolution  from  existing  PoRs.  There  are  a  number  of  extremely  large 
programs  that  could  benefit  significantly  in  cost,  performance  and  schedule  if  they  adopted 
“cloud”  technologies.  The  issue  is  that  they  are  large  PoRs  with  very  sizable  contractual 
commitments  and  are  either  unwilling  or  unable  to  embrace  the  opportunities.  An  example  of  this 
is  the  JMS  program.  Current  cost  of  the  system  is  approaching  $750M  with  no  end  in  sight. 
Implementing  a  “cloud-based”  solution  could  be  done  for  approximately  $125M  in  24  months 
with  senior  leader  advocacy. 


94 


Evolution  Roadmap.  The  DNI  has  developed  a  strategy.  The  DoD  CIO  has  a  strategy  as 
does  each  IC  agency.  The  challenge  is  that  implementation  is  currently  being  downward  directed 
and  the  path  to  interconnected  systems  is  so  dynamic  that  developers  are  having  a  difficult  time 
staying  in  synch  with  the  enterprise. 

Expert  5  Response 

DoD  funding,  DoD  cost  for  SIPR  solutions  (note  that  there  are,  I  believe,  two  commercial 
SIPR  cloud  providers,  but  will  be  in  direct  competition  with  DISA). 

Unbiased  cloud  service  brokering  processes. 

Requirements  gathering  and  vetting  and  ultimately  translated  into  the  right  service,  right 
vendor,  right  cost,  etc. 

The  ability  to  implement  private,  public,  and  hybrid  cloud  solutions  across  2  security 
enclaves  (NIPR  &  SIPR)  and  the  need  for  a  JWICs  solution. 

Expert  6  Response 

The  main  obstacles  are  unclear  /  ambiguous  requirements,  an  unsupportive  customer,  a 
technically  immature  environment  (both  environment  and  IT  IQ),  and  lack  of  overall 
organizational  support. 

Expert  7  Response 

The  expectation  of  a  vast  monetary  ROI  from  cloud  computing  is  an  illusion.  The  proper 
measure  of  a  successful  implementation  is  VALUE  and  how  value  should  be  measured  becomes 
the  key  critical  driver.  Cloud  computing  is  delivering  a  service  and  to  be  successful  the  service 
must  be  of  value  to  the  customer.  The  truth  of  the  matter  is  cloud  services  could  cost  more  to 
deliver  than  the  legacy  system  it  is  replacing.  However,  if  the  service  provided  is  valuable  to  the 
customers  using  it,  the  environment  will  deliver  value  through  higher  rates  of  usage,  productivity 
improvements  and  other  “soft”  benefits.  Classic  ROI  does  not  do  well  in  a  cloud  environment. 
Some  leaders  will  not  understand  that. 

Expert  8  Response 

Customer/client  actually  knowing  what  they  want/what  they  expect  for  the  price/level 
of  effort  and  to  establish  long  term  partnerships  to  allow  for  continued 
growth/effectiveness/efficiencies.  Many  times  the  client/customer  is  not  sure  of  what  they 
currently  have  and  therefore  not  sure  of  the  business  case  for  change-  how  do  they  ensure  they 
get  what  they  need  and  do  so  in  a  more  cost  efficient,  reliable  manner-  with  the  level  of 
security  and  future  capability  that  is  required. 
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EXPERT  QUESTION  4: 


What  are  the  differences  between  the  way  industry  implements  a  cloud  solution  to  the 

way  DoD  should? 

Expert  1  Response 

The  DOD  is  not  risk  averse,  its  change  averse...  otherwise  we  wouldn’t  accept  all  of  the 
risks  associated  with  our  current  IT  architectures.  Industry  is  far  better  at  managing  risk,  when  it 
comes  to  IT  services  than  the  DOD.  We  often  eschew  private  enterprise  because  of  their  desire 
for  profit  and  we  assume  that  they  “cut  comers”  on  security.  I  have  found  just  the  opposite  to  be 
true  for  many  established  cloud  computing  vendors.  In  the  cloud  computing  space,  there  are  not 
significant  barriers  preventing  consumers  from  switching  from  one  provider  to  another.  A 
sufficient  security  breech  could  result  in  a  tremendous  loss  of  revenue  as  consumers  make  that 
change.  These  financial  realities  make  security  a  top  priority. 

Expert  2  Response 

Industry  tends  to  run  on  a  best  effort.  The  DoD  would  require  stability,  security,  and 
compartmentalization  of  the  cloud  in  order  to  protect  assets  and  information. 

Many  DOD  projects  do  not  just  have  an  external  security  concern,  but  also  an  internal 
one.  Since  I  do  development,  what  I  run  on  my  servers  is  something  that  I  should  not  have  to  go 
through  a  huge  effort  to  get  onto  the  cloud  servers.  When  I  have  access  to  a  rack  of  servers, 
correctly  separated  from 

the  rest  of  the  network,  then  I  can  put  software  on  it  during  development,  but  there  would  seem  to 
be  issues  in  doing  that  with  a  shared  resource  in  a  DoD  environment.  Also,  it  is  not  the  case  that 
development  can  be  done  on  a  noncloud  computing  environment,  approved  (getting  an  ATO)  and 
then  putting  it  on  the  cloud. 

The  cost  allocation  for  computational  resources  is  also  very  different.  A  commercial 
project  or  a  DoD  contract  project  sets  aside  a  certain  amount  of  ODCs  and  they  are  used  to  buy 
equipment.  Using  a  cloud  computing  solution  confuses  the  situation  since  they  are  closer  to 
services,  and  so  there  needs  to  be  a  shift  in  the  way  that  the  money  is  spent,  and  project  managers, 
COTRs,  contracting  officers  need 
to  understand  it. 

Expert  3  Response 

Industry  has  much  larger  numbers  of  companies  to  build  cloud  applications  and  these 
companies  have  incentives  to  do  so.  DoD  has  a  relatively  tiny  number  and  these  companies  have 
no  incentive  other  than  the  labor  dollars  they  will  get  paid  to  do  so.  Development  labor  is  a  small 
fraction  of  the  revenue  of  these  companies  -  so  an  environment  that  reduces  their  O&M  revenue 
will  make  cloud  computing  very  difficult  for  these  companies  to  truly  get  on  board  with. 

Expert  4  Response 

Sense  of  urgency.  Time  is  money  in  the  business  world.  If  a  solution  can  save  money 
while  simultaneously  saving  time,  they’re  much  more  willing  to  take  the  risks.  Additionally, 
businesses  are  constantly  challenged  by  their  competitors  so  they  don’t  have  the  luxury  of 
exhaustively  studying  their  options  like  the  DoD. 
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Risk.  Industry  tends  to  build  a  little,  test  a  little  and  adjust  the  course  of  development. 
DoD,  on  the  other  hand,  spends  so  much  time  investigating  the  options  that  by  the  time  a 
requirements  document  is  presented  for  bids,  the  requirements  will  frequently  be  unable  to  reflect 
the  current  state  of  technology.  There  is  some  merit  to  this  because  DoD  generally  measures  risk 
by  lives  saved  versus  dollars  spent  but  it’s  almost  to  the  point  of  stagnating  progress. 

Leadership.  Most  military  leaders  have  not  been  promoted  for  being  innovative.  Just  as 
with  our  federal  bureaucracy,  change  comes  very  slowly.  Commercial  adoption  of  technology  is 
driven  by  market  share  and  business  leaders  recognize  that  if  they  don’t  move,  they  may  not  be 
here  tomorrow. 

Expert  5  Response 

From  the  Operations  Community  is  maintaining  situational  awareness  to  the  granularity 
the  DoD  has  grown  to  expect. 

Actual  service  level  provided  vice  service  level  paid  for.  For  example,  DISA  hosts 
applications  in  the  Defense  Enterprise  Computing  Centers  (DECCs),  and  these  systems  have  a 
Mission  Assurance  Category  (MAC)  applied  to  them  which  has  an  associated  cost.  If  the  Marine 
Corp  Online  systems  goes  down,  a  general  officer  to  general  officer  call  is  made,  and  it  is  treated 
as  a  MAC  I  (mission  critical,  restoration  time  within  a  few  hours,  etc)  even  though  the  USMC 
isn’t  paying  for  that  level  of  service  (which  has  no  MAC  associated  with  it,  which  means  we  have 
say  4  days  to  address  the  problem). 

DoD  has  different  security  rules/processes  than  commercial  providers,  and  I  think  ours 
tend  to  be  more  strict,  which  may  drive  up  our  cost,  increase  implementation  time,  etc. 

Traditionally,  the  DoD  isn’t  about  making  money  and  some  of  our  organizations  are 
inflated  due  to  geographic  location.  This  drives  the  price  up,  and  I  think  it  will  be  hard  for  us  to 
compete  financially  with  commercial  providers. 

DoD  often  has  complicated  ATO  and  DAA  processes  to  operate  systems  on  a  DoD 
network.  The  commercial  sector  may  not  have  this  issue.  It  certainly  affects  our  ability  to  be  agile 
around  customer  needs. 

Expert  6  Response 

The  DoD  has  been  adopting  capabilities  in  a  manner  that  Industry  has  been  delivering 
them.  The  DoD  use  cases  and  implementation  are  different  than  the  Industry  use  cases.  In 
addition,  the  emphasis  on  C&A,  1A  controls,  and  ATO’d  environments  delay  most  if  not  timely 
implementations. 

Expert  7  Response 

DoD  seems  to  acquire  a  solution  before  they  have  defined  the  problem  it’s  supposed  to 
solve.  Industry  will  define  the  problem  from  various  perspectives  and  then  explore  solutions  that 
deliver  the  highest  value  for  the  most  reasonable  price. 

Expert  8  Response 

Commercial  industry  finds  a  long  term  partner  who  is  a  proven  expert  in  the  field  (has  the 
experience  and  the  capability),  hands  over  the  mission,  looks  for  long  term  associations  where 
benefits  are  shared  (e.g.  the  commercial  provider  saves  the  client  money  and  makes  money  from 
that  savings-  direct  benefits  from  quality  work  and  providing  efficiencies  and  improved 
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performance).  Commercial  clients  focus  on  the  SLAs  and  not  what  is  behind  the  door-  the 
reason  they  hired  an  expert  to  do  the  work.  DoD  has  a  since  of  doubt-  they  must  check  and 
control  everything  vs.  a  managed  services  approach  based  on  delivery  of  a  product  of  the  quality 
and  capability  required  and  at  a  price  incentivized  by  performance  and  efficiency/cost  savings. 


EXPERT  QUESTION:  5 

How  could  those  obstacles  and  differences  be  overcome? 

Expert  1  Response 

Embrace  sequestration!  Our  fiscally  constrained  environment  will  force  us  to  chart  new 
territory  (at  least  new  to  the  DOD).  The  good  news  is  that  we  have  many  examples  of  how  private 
industry  is  successfully  leveraging  cloud  implementations  to  their  great  benefit.  All  we  need  to  do 
is  follow.  That’s  hard  for  us,  especially  since  we  are  the  industry  driver  in  most  other  aspects  of 
the  DOD  (fighter  airplanes,  most  satellites,  tanks,  etc).  This  isn’t  so  in  the  information  technology 
space.  We  are  a  drop  in  the  overall  bucket.  For  example,  according  to  DARPA’s  website 
(www.darpa.mil/WorkArea/DownloadAsset.aspx?id=2 147486441),  their  2014  budget  submission 
was  ~$2.8B  and  depending  on  how  you  slice  and  dice  their  program  lines,  only  about  a  third  of 
that  is  IT  related.  Now,  look  at  just  Microsoft’s  research  and  development  budget  for  the  period 
ending  in  June  2012.  Microsoft  spent  almost  $  1  OB !  That’s  just  one  company.  Google  spends  a 
little  less  than  $2B  annually  and  Intel  spends  around  $2.5B.  We  simply  cannot  compete  with 
those  numbers  and  expect  to  influence  the  direction  of  the  IT  market  through  direct  investment. 
We  really  need  to  figure  out  how  to  get  on  this  train  rather  than  complain  about  not  building  the 
tracks. 

Expert  2  Response 

There  will  need  to  be  a  development  cloud  (with  development  network  or  unconnected 
connectivity)  and  an  Approved  To  Operate  cloud.  There  will  need  to  be  a  learning  process  on 
how  to  write  the  cloud  computing  resources  into  RFPs,  proposals,  and  contracts.  Cloud  resource 
providers  will  need  to  ensure  that  resources  are  sufficient  to  provide  the  desired  computational 
resources  (and  memory  and  disk  space)  to  satisfy  the  users.  It  will  not  take  many  instances  of 
someone  unable  to  get  the  machines  that  they  need  before  they  will  give  up  and  simply  buy  their 
own  machines. 

Expert  3  Response 

I'm  not  sure  how  to  overcome  these  obstacles.  The  incentive  structure  probably  has  to 
somehow  be  changed. 

Expert  4  Response 

Implement  aggressive  timelines  with  manageable  and  measurable  milestones  to  deliver 
real  capabilities  incrementally.  Don’t  try  to  build  the  whole  house  at  once. 

Seek  out  the  little  guys.  There  are  hundreds  of  very  small  businesses  that  are  hungry  to 
deliver  capabilities  very  cheaply  and  very  quickly.  1  don’t  think  anyone  would  equate  the  large 
DoD  corporations  as  agile  or  nimble  nor  are  they  willing  to  sign  a  contract  for  six  months  for 
$200k.  That’s  not  how  they  became  multi-billion  dollar  companies.  That’s  how  you  build  an  F- 
35. 
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Simplify  scope  and  requirements.  Again,  build  a  little  and  test  a  little  to  build  momentum. 
Expert  5  Response 

The  DoD  needs  to  get  out  of  the  Cloud  business,  and  offer  it  as  a  contract  service.  Don’t 
try  to  compete  by  providing  a  private  cloud  on  NIPR  and  all  classes  of  clouds  on  SIPR.  DoD 
(DISA)  should  establish  security  and  NetOps  criteria  and  simply  broker  the  services  (have 
contracts  available  for  users,  and  assist  in  matching  user  requirements  to  providers). 

Expert  6  Response 

DoD  partners  and  agencies  should  partner  together  to  build  a  community  that  delivers 
cloud  capabilities  that  can  be  easily  shared  for  maximum  reuse.  The  community  should  also 
engage  in  following  community  practices  for  ATO  for  ‘type  accreditation’  where  any 
organization  and  agency  that  is  developing  capabilities  that  achieves  accreditation  should  be  able 
to  apply  these  capabilities.  Unfortunately,  every  agency  has  its  own  rules  for  C&A  but  should 
honor  a  community  accreditation  approval. 

Expert  7  Response 

In  my  view  DoD  is  moving  in  the  right  direction.  The  obstacles  faced  by  DoD  fall  for  the 
most  part  in  thinking  and  behaving.  The  business  enterprise  of  DoD  is  not  that  different  from  a 
large  multinational/global  company.  Both  must  have  HR,  Logistics,  Finance,  Infrastructure,  and 
other  common  business  functions.  DoD  has  a  culture/people/institutional  barrier  to  think  and 
behaving  as  a  business  enterprise  in  areas  where  that  think  could  benefit  them  the  most.  If  you 
want  to  be  an  enterprise,  think  and  behave  as  one. 

Expert  8  Response 

Managed  services  approach  vs.  buying  individual  capability,  long  term  contracts  with  clear  exit 
clauses  for  non-performance,  commercially  based  SLAs,  and  standard  levels  of  service  across  the 
organizations  vs.  constantly  treating  every  org  is  a  one-off. 
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Part  B  : 


This  section  contains  three  items:  first,  a  table  listing  what  we  think  may  be  the  main 
seven  drivers  of  Cloud  Implementation  success,  based  on  the  responses.  Second,  a  model 
showing  how  these  items  may  lead  to  success.  Third,  a  process  model  showing  how  success  is 
bred,  and  the  general  order  of  events  in  which  the  task  progresses.  The  instructions  for  each  of 
these  sections  is  listed  at  the  top  of  each  section.  As  you  accomplish  this,  keep  in  mind  the  task: 
determine  what  will  cause  "successful  cloud  computing  implementation  in  the  Air  Force. " 

Bl.  These  tables  contain  categorized  causes  based  on  expert  responses.  They  also  contain  inputs 
that  were  identified  to  support  the  cause  (hyperlinked  to  where  they  were  mentioned  within  the 
responses,  by  ctrl-click  you  can  view  an  elaboration  about  the  input).  Once  validated  by  the 
experts  these  causes  will  be  incorporated  into  the  model.  Please  comment  on  the  following: 

For  each  of  the  sub-items,  please  note  whether  you  feel  it  will  positively  effect  cloud  computing 
success  (+),  negatively  effect  (-),  or  perhaps  both  (+/-),  and  add  explanations  as  desired. 

Rank  each  cause  from  1-7  for  successful  cloud  computing  implementation  in  the  Air  Force. 
EXTRA  CREDIT.  Do  you  agree  with  the  categories  and  the  items  they  contain?  If  not,  please 
modify  them  or  create  your  own  categorization. 

EXTRA  CREDIT.  Did  we  miss  something,  or  is  there  anything  you  would  like  to  add  based  on 
review  of  each  others’  comments  (written  comments)? 


Cause 

Supporting  Inputs  (References  to  expert 
responses) 

Pos.  or 

Neg.  Effect 

Effect 

Access 

? 

Cloud 

computing 

success 

Aggregate  Programs  into  the  cloud 

Limit  Restrictions 

Access 

Data  access 

Sharing  Resources 

Min.  barrier  to  entry 

Application  restrictions 

Ability  to  implement  across  NIPR  and  SIPR 

DoD  reauires  compartmentalization 

Computational  services 

Ensure  sufficient  resources 

Managed  service  approach  vs.  buying 

individual  capability 
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Cause 

Supporting  Inputs  (References  to  expert 
responses) 

Pos.  or  Neg. 
Effect 

Effect 

Reliability 

? 

Cloud 

computing 

success 

Define  Intangibles  (Scalability  and 

Flexibility) 

Interconnectivitv 

Reliability 

Performance 

Interconnectivitv 

Levels  of  performance  expectations 

DoD  requires  stability 

Cause 

Supporting  Inputs  (References  to  expert 
responses) 

Pos.  or 

Neg. 

Effect 

Effect 

Budget 

Cloud 

computing 

success 

Sufficient  Time  for  Investment 

Cost 

Min.  overhead 

Government  Budget  Cvcle 

DoD  funding 

Illusion  of  a  vast  monetary  ROI 

Prices  driven  up  due  to  inflated  units  because  of 

geographic  location 

Leverage  a  larger  IT  budget 

Long  term  contracts  with  exit  clauses  for  non- 

performance 
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Cause 


Supporting  Inputs  (References  to  expert 
responses) 


Understanding 


Incentive  to  use  cloud  computing 


Common  Understanding 

User  understanding 

Recognize  the  business 

aspects. .  .productivity  improvements. 

lowered  total  cost  of  ownership 

and  hard  dollar  savings 

Understanding  and  definition 

Manager  Understanding  of  metrics 

Weak  support  model 

Not  everything  belongs  in  the  cloud 

Discovery. .  .what  is  present  and  where  to 

go 

Outdated  Acquisition  Mindset 

Cvber  Threat  "Awareness"  vs 

"Understanding" 

Inability  to  auantifv  gains 

Old  think 

Unbiased  brokering  service 

Unclear/ambiguous  requirements 

Leaders  not  understanding  the  benefits 

short  of  a  ROI 

Customer  knowing  what  they 

want. .  .reliability,  security,  efficiencies. 

effectiveness 

Leverage  Cloud  Benefits 

Learning  process  to  write  RLPs,  proposals. 

and  contracts 

Change  the  incentive  structure 

Implement  aggressive  timelines 

Simplify  scope  and  requirements 

If  DoD  wants  to  be  an  enterprise  then  think 

and  behave  as  one 


Pos.  or 

Neg. 

Effect 


9 


Effect 

Cloud 

computing 

success 
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Cause 

Supporting  Inputs  (References  to  expert 
responses) 

Pos.  or 

Neg. 

Effect 

Effect 

Security 

? 

Cloud  computing 
success 

Trust 

Security 

Security 

Security  priority 

DoD  reauires  security 

Risk 

Different  security  protocols  in  DoD 

Cause 


Supporting  Inputs  (References  to  expert 
responses) 


Standards 


No  standard  between  DoD  agencies 


No  standard  SLAs  that  provide  "best  value" 

Policy 

Conflict  of  interest.  DISA  is  the  cloud  broker 

and  cloud  provider 

Outdated  Policies 

Acquisition  process 

Migration/evolution  from  existing  PoRs  (??) 

Evolution  Roadmap 

Provided  service  level  vs.  service  level  paid 

for 

Complicated  ATO  and  DAA  process  to  be  on 

networks 

DoD  use  cases  and  implementation  are 

different 

C  &  A,  IA  controls,  and  ATO'd  environments 

delay  implementation 

Industry  focuses  on  SLAs  not  what  lies 

"behind  the  door" 

Development  cloud  and  an  ATO  cloud 

Agencies  honor  community  C  &  A 

accreditation  approval 

Commercial  based  SLAs 

Standard  levels  of  service 

Pos.  or 

Neg. 

Effect 


? 


Effect 

Cloud 

computing 

success 
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Cause 


Supporting  Inputs  (References  to  expert 
responses) 


Environment 


Cloud  as  a  Business  Process 


"Good"  customer  environment 

Environment  supports  implementation,  min. 

environmental  inertia,  and  technical  10  of 

staff 

Excessive  beaurocracv  creating  excessive 

timeline 

Innovative  applications 

Adeauate  advertising/marketing 

DISA  reputation 

Lack  of  "good"  customer  base  for 

implementation  and  utilization 

Organizational  chum 

Mature  business  model/enterprise  business 

model 

Service  advocacy 

Unsupportive  customer 

Technically  immature  environment 

Lack  of  organizational  support 

DoD  is  change  adverse 

Industry  uses  best  effort 

Incentives  for  industry  to  build  applications 

Sense  of  urgency 

Leadership 

Maintain  situational  awareness  to  the  extent 

that  DoD  has  grown 

DoD  acquires  solution  before  defining  the 

problem 

Industry  defines  problem  from  various  views 

that  provide  highest  value 

Industry  uses  long  term  partners  with 

required  expertise 

DoD  doubt.... they  must  "check  and  control 

everything" 

Seek  out  the  little  guvs 

DoD  needs  to  get  out  of  cloud  business  and 

offer  cloud  as  contract  service 

DoD  partners  and  agencies  should  partner 

together  to  deliver  cloud  cap.  for  max  reuse 

Pos.  or 

Neg. 

Effect 


? 


Effect 

Cloud 

computing 

success 
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B2.  A  “variance”  model  is  a  “cause  and  effect”  model.  It  provides  an  explanation  of  how  input 
variables  “cause”  an  output — or  dependent — variable  (Cloud  Computing  Implementation 
Success,  labeled  as  the  “DV”  in  the  model).  These  factors  can  have  a  positive  or  negative  effect 
on  that  variable,  also  any  number  of  additional  factors  could  have  an  effect  on  the  output  variable. 
This  type  of  model  can  be  translated  into  a  mathematical  equation  (Y  =  X  +  Z  +  . . .  n). 

Do  you  agree  with  the  categories/model?  If  not,  please  feel  free  to  produce  your  own  model  of 
the  type  shown,  or  use  pen-and-ink  to  alter  it  in  a  way  that  makes  more  sense  from  your 
perspective. 

For  each  of  the  input  variables,  please  note  whether  you  feel  it  will  positively  effect  cloud 
computing  success  (+),  negatively  effect  (-),  or  perhaps  both  (+/-),  and  add  explanations  as 
desired  (feel  free  to  write  the  symbols  next  to  the  arrows  on  the  diagram). 

EXTRA  CREDIT.  Is  there  anything  you  would  like  to  add  based  on  review  of  each  other’s 
comments  (written  comments)? 
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B3.  A  process  model  provides  a  rational  explanation  of  processes  by  evaluating  possible  courses 
or  paths  based  observed  factors.  It  provides  "linkage"  between  those  factors,  where  one  needs  to 
be  fulfilled  before  moving  to  the  next  until  the  model  is  complete.  Also,  process  models  may 
contain  feedback  loops  between  any  identified  factors  within  the  model. 

Do  you  agree  with  the  categories/model?  If  not,  please  feel  free  to  produce  your  own  model  of 
the  type  shown,  or  use  pen-and-ink  to  alter  it  in  a  way  that  makes  more  sense  from  your 
perspective. 

EXTRA  CREDIT.  Is  there  anything  you  would  like  to  add  based  on  review  of  each  other’s 
comments  (written  comments)? 
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CLOUD  COMPUTING  IMPLEMENTATION  ORGANIZATIONAL  SUCCESS  IN  THE 

AIR  FORCE 

Delphi  3rd  Pass 

Greetings,  welcome,  and  again  thank  you  for  participating!  This  the  3rd  and  last  round  of  the 
Delphi  study.  In  this  round,  you  are  given  the  final  models  to  review.  What  we  ask  in  this  round 
is  that  you  review  the  models  and  select  concur  or  concur  with  comments  in  the  identified 
section.  Depending  on  the  desired  level  of  effort,  this  round  should  take  no  more  than  15 
minutes.  In  the  interest  of  preserving  time,  no  response  will  be  taken  as  a  concurrence  to  both 
models. 

We  appreciate  your  patience  as  we  analyzed  the  data  from  Round  1  and  Round  2.  Upon 
completion  of  Round  3,  these  finalized  model(s)  will  be  represented  in  the  research  thesis — 
essentially  a  confirmation  that  we  have  converged  on  final  models.  Feel  free  to  ask  for 
explanation  on  anything  you  do  not  understand. 

As  before,  all  data  will  be  aggregated  or  otherwise  processed  before  public  release.  Names  and 
associated  data  will  be  protected  at  all  times,  only  known  to  the  researchers,  and  managed 
according  to  the  AFIT  interview  protocol.  All  data  will  only  be  handled  by  the  researcher  and 
advising  committee.  Please  feel  free  to  contact  me  with  any  questions  at  937-541-8169.  Provide 
responses  to  corey.perkins@us.af.mil  NLT  15  November,  2013. 
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Part  A  : Variance  Model 


A  “variance”  model  is  a  “cause  and  effect”  model.  It  provides  an  explanation  of  how 
input  variables  “cause”  an  output — or  dependent — variable  (Cloud  Computing  Implementation 
Success,  labeled  as  the  “DV”  in  the  model).  These  factors  can  have  a  positive  or  negative  effect 
on  that  variable,  also  any  number  of  additional  factors  could  have  an  effect  on  the  output  variable. 
This  type  of  model  can  be  translated  into  a  mathematical  equation  (Y  =  X  +  Z  +  . . .  n). 


Please  select  option  and  input  comments  as  needed. 

r  Concur 

Concur  with  Comments 
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Part  B  :  Process  Model 


A  process  model  provides  a  rational  explanation  of  processes  by  evaluating  possible 
courses  or  paths  based  observed  factors.  It  provides  "linkage"  between  those  factors,  where  one 
needs  to  be  fulfilled  before  moving  to  the  next  until  the  model  is  complete.  Also,  process  models 
may  contain  feedback  loops  between  any  identified  factors  within  the  model. 


Please  select  option  and  input  comments  as  needed. 

Concur 

Concur  with  Comments 
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Appendix  E:  IRB  Waiver  Request 


6  May  2013 


MEMORANDUM  FOR  711  HPW/IR  (AFRL  IRB) 

FROM:  AFIT/ENV  (Lt  Col  Darin  Ladd) 

2950  Hobson  Way 
WPAFB  OH,  45433-7765 

SUBJECT:  Request  for  exemption  from  human  experimentation  requirements  (32  CFR  219,  DoDD  3216.2 
and  AFI 40-402)  for  cloud  computing  Delphi  study. 

1.  This  exemption  request  is  based  on  the  Code  of  Federal  Regulations,  title  32,  part  219,  section  101, 
paragraph  (b).  The  following  information  is  provided  to  show  cause  for  this  exemption. 

2.  Purpose.  The  purpose  of  this  study  is  to  ask  experts  questions  regarding  cloud  computing 
implementation.  The  questions  will  be  asked  using  the  Delphi  research  method  utilizing  multiple  passes  to 
formulate  accurate  results.  Aggregated  results  and  sporadic  quotations  may  be  published  in  a  peer-reviewed 
project  management  outlet. 

3.  Subjects.  The  subjects  chosen  for  these  questions  are  experts  in  DoD  and  throughout  industry. 
Demographic  information  is  not  available  for  these  subjects,  and  random  sampling  is  not  used  to  identify 
participants — although  snowball  sampling  may  be  used.  The  actual  number  of  subjects  will  vary;  however, 
between  7  and  10  individuals  may  participate. 

4.  Timeframe:  May  15,  2013  -  March  1,  2014 

5.  Data  collected:  This  study  will  collect  specific  demographic  information,  but  only  generalities  will  be 
used  when  the  data  is  presented.  The  only  information  collected  from  participants  is  that  at  attachments 
(Tabs  1-2).  All  data  will  be  aggregated  or  otherwise  processed  before  public  release.  Interview  consent 
forms  will  be  stored  separate  from  interview  responses,  reducing  the  possibility  that  responses  may  be 
linked  with  individual  participants.  Answers  will  be  stored  digitally  to  allow  the  researcher  to  properly 
review  responses  (especially  quotes).  Responses  will  be  kept  for  the  required  24  months,  and  then 
destroyed.  I  understand  that  the  names  and  associated  data  I  collect  must  be  protected  at  all  times,  only  be 
known  to  the  researchers,  and  managed  according  to  the  AFIT  interview  protocol.  All  data  will  only  be 
handled  by  the  following  researcher  and  advising  committee: 

Researcher/Student:  MSgt  Corey  J.  Perkins 

Primary  Investigator/Committee  Chair:  Lt  Col  Darin  A.  Ladd 

Committee  Members:  Lt  Col  Brent  T.  Langhals  and  SMSgt  Jeffrey  C.  Sandusky 

6.  Risks  to  Subjects:  The  primary  risk  is  disclosure  of  individual  responses  or  private  information.  In  the 
case  of  quotations,  it  may  be  possible  for  a  reader  to  identify  an  individual  based  on  his/her  quotation.  For 
this  reason,  the  researcher  will  allow  the  participants  to  review  and  redact  their  quotations,  if  desired.  The 
subjects  discussed  are  lawful  and  mostly  non-sensitive;  however,  the  participants  are  more  senior  leaders 
for  whom,  if  their  candid  responses  were  linked  to  their  person,  could  cause  discomfort  but  likely  not 
adverse  action.  If  a  subject’s  future  response  reasonably  places  them  at  risk  of  criminal  or  civil  liability  or 
is  damaging  to  their  financial  standing,  employability,  or  reputation,  I  understand  that  I  am  required  to 
immediately  file  an  adverse  event  report  with  the  IRB  office. 

7.  Informed  consent:  All  subjects  voluntarily  participate.  No  adverse  action  is  taken  against  those  who 
choose  not  to  participate.  Subjects  are  made  aware  of  the  nature  and  purpose  of  the  research,  sponsors  of 
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the  research,  and  disposition  of  the  survey  results  using  the  attached  consent  form  (Tab  4).  A  copy  of  the 
Privacy  Act  Statement  of  1974  is  presented  for  their  review. 

8.  Please  feel  free  to  contact  me  with  any  questions  at  785-6565,  x4228,  or  darin.ladd@afit.edu. 


DARIN,  A.  Ladd,  Lt  Col,  Ph.  D„  USAF 
Director,  Communications  &  Information,  AFIT 


2  Attachments: 

1 .  Delphi  Questions 

2.  Delphi  Consent  Form 
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Research  Questionnaire 


Tab  1:  Delphi  Questions 


Proposed  questions  to  experts: 

EXPERT  QUESTION  1: 

What  factors  should  be  considered  for  successful  cloud  computing  implementation? 

EXPERT  QUESTIONS  2: 

What  are  key  concerns  for  successful  cloud  computing  utilization? 

EXPERT  QUESTION  3: 

What  main  obstacles  would  hinder  successful  cloud  computing  implementation? 

EXPERT  QUESTION  4: 

What  are  the  differences  between  the  way  industry  implements  a  cloud  solution  to  the  way 
DoD  should? 

EXPERT  QUESTION:  5 

How  could  those  obstacles  and  differences  be  overcome? 

FOLLOW  UP  QUESTION: 

Is  there  anything  else  not  covered  that  could  be  added? 
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Tab  2:  Delphi  Consent  Form 


6  May,  2013 

Greetings!  You  are  being  asked  to  take  part  in  a  research  study  carried  out  by  MSgt  Corey  J.  Perkins,  a 
student  at  the  Air  Force  Institute  of  Technology  (AFIT).  This  form  explains  the  study  and  your  part  in  it  if 
you  decide  to  join.  Please  read  the  form  carefully.  Feel  free  to  ask  for  explanation  on  anything  you  do  not 
understand.  If  you  join  the  study,  you  can  change  your  mind  later  or  quit  at  any  time,  without  any  penalty 
or  loss  of  services  or  benefits. 

Study  Title:  Cloud  Computing  Implementation  Organizational  Success  In  The  Air  Force. 

Primary  Researcher: 

Name  Title/Department  E-mail  Telephone 

Darin  Director,  Communications  &  Darin.ladd@afit.edu  DSN:  785-6565,  x4228 

Ladd  Information,  AFIT 

What  is  this  study  about?  This  research  study  investigates  the  factors  that  will  affect  the  successful 
implementation  of  cloud  computing  Software  as  a  Service  in  the  Air  Force  by  the  Defense  Information 
Service  Agency  (DISA).  You  are  being  asked  to  take  part  in  this  study  because  you  are  in  a  position  to 
answer  the  q  questions  of  interest.  Taking  part  in  the  study  will  take  about  30-45  minutes  per  session 
over  3-4  sessions. 

What  will  I  be  asked  to  do  if  I  am  in  this  study?  If  you  take  part  in  the  study,  you  will  be  asked  to 
answer  five  questions  in  multiple  "passes."  Each  pass  should  not  take  more  that  30-45  minutes  of  your 
time.  The  passes  will  be  conducted  electronically  via  e-mail  with  the  researcher.  On  the  first  pass  experts 
will  be  given  two  weeks  to  answer  a  set  of  pre -determined  "vague"  questions  that  will  be  asked  to  give  the 
experts  a  chance  to  elaborate  their  answers.  Once  the  answers  are  received,  they  are  collected,  analyzed  and 
like  factors  will  be  grouped  together.  One  week  later  a  summary  of  the  collected  responses  will  be  sent  to 
the  panel  for  further  clarification  regarding  their  view  on  validity  of  the  first  responses.  The  experts  will 
once  again  get  two  weeks  to  respond  to  the  "second  pass."  Within  one  week  this  additional  data  will  be 
consolidated  into  a  model  containing  the  identified  factors.  On  the  "third  pass"  the  model  is  sent  back  out 
to  the  panel  for  validity  and  comment,  once  again  it  should  be  returned  within  a  two  week  period.  This  last 
round  of  inputs  from  the  panel  will  further  enhance  the  model  for  the  appropriateness  of  the  Measures  of 
Effectiveness  (MoEs).  All  three  passes  should  be  complete  by  30  July,  2013. 

Are  there  any  benefits  to  me  if  I  am  in  this  study?  Though  experts  will  not  be  identified,  the  information 
collected  pertinent  to  the  research  will  be  encapsulated  in  a  thesis.  This  thesis  will  be  presented  to  DISA 
and  will  emphasize  the  areas  to  be  focused  during  implementation.  This  model  will  evolve  and  could  be 
provided  as  guidance  to  industry  vendors  that  are  outsourced  by  DISA  for  the  cloud  solution.  That  insight 
will  allow  industry  vendors  to  provide  a  valuable  and  effective  solution. 

Are  there  any  risks  to  me  if  I  am  in  this  study?  Because  this  research  requests  your  subjective  opinion 
regarding  success  factors,  some  of  the  information  sought  might  be  considered  sensitive,  and  may  cause 
discomfort.  For  this  reason,  you  may  refuse  to  answer  any  question  at  any  time,  and  likewise  may  stop  at 
any  time.  The  answers  will  be  stored  digitally,  with  all  identifying  information  stripped  from  them.  It  is 
important  to  note  that  it  might  still  be  possible  for  a  reader  of  the  final  written  product  to  attribute  results  to 
a  given  individual  and/or  organization.  You  will  be  given  an  opportunity  to  review  this  information  and 
make  a  reasoned  judgment  of  the  risks  of  divulging  such  information. 

Will  my  information  be  kept  private?  The  data  for  this  study  will  be  kept  confidential  to  the  extent 
allowed  by  federal  and  state  law.  No  published  results  will  identify  you,  and  your  name  will  not  be 
associated  with  the  findings.  Under  certain  circumstances,  information  that  identifies  you  may  be  released 
for  internal  and  external  reviews  of  this  project.  The  digital  file  containing  the  responses,  as  well  as  the 
study  write-up  will  be  secured.  Your  information  will  only  be  released,  if  requested,  to  authorized  members 
of  the  AFIT  Institutional  Review  Board,  to  ensure  research  compliance  with  federal  and  state  law.  Your 
information  will  not  be  released  to  any  other  entity.  The  results  of  this  study  may  be  published  or  presented 
at  professional  meetings,  but  the  identities  of  all  research  participants  will  remain  anonymous.  The  data  for 
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this  study  will  be  kept  as  required  by  AFIT  policy,  after  which  time  the  digital  file  containing  the  interview 
will  be  destroyed. 

Are  there  any  costs  or  payments  for  being  in  this  study?  There  will  be  no  costs  in  this  study. 

Who  can  I  talk  to  if  I  have  questions?  If  you  have  questions  about  this  study  or  the  information  in  this 
form,  please  contact  the  researcher  using  the  contact  information  provided  above.  If  you  have  questions 
about  your  rights  as  a  research  participant,  or  would  like  to  report  a  concern  or  complaint  about  this  study, 
please  contact  the  WPAFB  Institutional  Review  Board  at  (937)  255-3636,  x4543  or  e-mail 
HumanSubjects@afit.edu,  or  regular  mail  at:  Wright  Research  Site  IRB,  711  HPW/IR,  2245  Monahan 
Way,  Wright-Patterson  AFB,  OH  45433 

What  are  my  rights  as  a  research  study  volunteer?  Your  participation  in  this  research  study  is 
completely  voluntary.  You  may  choose  not  to  be  a  part  of  this  study.  There  will  be  no  penalty  to  you  if 
you  choose  not  to  take  part.  You  may  choose  not  to  answer  specific  questions  or  to  stop  participating  at 
any  time. 

What  does  my  signature  on  this  consent  form  mean?  Your  signature  on  this  form  means  that:  a)  you 
understand  the  information  given  to  you,  b)  you  have  been  able  to  ask  the  researcher  questions  and  state 
any  concerns,  c)  the  researcher  has  responded  to  your  questions  and  concerns,  d)  you  believe  you 
understand  the  research  study  and  the  potential  benefits  and  risks  involved. 


Statement  of  Consent:  I  give  my  voluntary  consent  to  take  part  in  this  study.  I  will  be  given  a  copy  of 
this  consent  document  for  my  records. 


Signature  of  Participant  Printed  Name  of  Participant  Date 


Statement  of  Person  Obtaining  Informed  Consent:  I  carefully  explained  to  the  person  taking  part  in  the 
study  what  he  or  she  can  expect.  I  certify  that  when  this  person  signs  this  form,  to  the  best  of  my 
knowledge,  he  or  she  understands  the  purpose,  procedures,  potential  benefits,  and  potential  risks  of 
participation.  I  also  certify  that  he  or  she:  a)  speaks  the  language  used  to  explain  this  research,  b)  reads 
well  enough  to  understand  this  form,  c)  does  not  have  any  problems  that  could  make  it  hard  to  understand 
what  it  means  to  take  part  in  this  research. 


Signature  of  Researcher 


Printed  Name  of  Researcher  Date 
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Appendix  F :  IRB  Waiver  Approval 


30  May  2013 


MEMORANDUM  FOR  LT  COL  DARIN  A.  LADD, 

FROM:  William  A.  Cunningham,  Ph.D. 

AFIT  IRB  Research  Reviewer 
2950  Hobson  Way 

Wright-Patterson  AFB,  OH  45433-7765 

SUBJECT :  Approval  for  exemption  request  from  human  experimentation  requirements 
(32  CFR  219,  DoDD  3216.2  and  AFI  40-402)  for  Cloud  Computing  Implementation 
Organizational  Success  In  The  Air  Force. 

Your  request  was  based  on  the  Code  of  Federal  Regulations,  title  32,  part  219,  section 
101,  paragraph  (b)  (2)  Research  activities  that  involve  the  use  of  educational  tests 
(cognitive,  diagnostic,  aptitude,  achievement),  survey  procedures,  interview  procedures, 
or  observation  of  public  behavior  unless:  (i)  Information  obtained  is  recorded  in  such  a 
manner  that  human  subjects  can  be  identified,  directly  or  through  identifiers  linked  to  the 
subjects;  and  (ii)  Any  disclosure  of  the  human  subjects’  responses  outside  the  research 
could  reasonably  place  the  subjects  at  risk  of  criminal  or  civil  liability  or  be  damaging  to 
the  subjects’  financial  standing,  employability,  or  reputation. 

Your  study  qualifies  for  this  exemption  because  you  are  not  collecting  sensitive  data, 
which  could  reasonably  damage  the  subjects’  financial  standing,  employability,  or 
reputation.  Further,  the  demographic  data  you  are  collecting  cannot  realistically  be 
expected  to  map  a  given  response  to  a  specific  subject. 

This  determination  pertains  only  to  the  Federal,  Department  of  Defense,  and  Air  Force 
regulations  that  govern  the  use  of  human  subjects  in  research.  Further,  if  a  subject’s 
future  response  reasonably  places  them  at  risk  of  criminal  or  civil  liability  or  is  damaging 
to  their  financial  standing,  employability,  or  reputation,  you  are  required  to  file  an 
adverse  event  report  with  this  office  immediately. 


WIFFIAM  A.  CUNNINGHAM,  PH.D. 
AFIT  Research  Reviewer 
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